Multiple camera system for wide angle imaging

ABSTRACT

Systems and techniques are described for large field of view digital imaging. A device&#39;s first image sensor captures a first image based on first light redirected from a first path onto a redirected first path by a first light redirection element, and the device&#39;s second image sensor captures a second image based on second light redirected from a second path onto a redirected second path by a second light redirection element. A virtual extension of the first path beyond the first light redirection element can intersect with a virtual extension of the second path intersect beyond the second light redirection element. The device can modify the first image and second image using perspective distortion correction, and can generate a combined image by combining the first image and the second image. The combined image can have a larger field of view than the first image and/or the second image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.63/040,661, filed Jun. 18, 2020 and titled “Multiple Camera System forWide Angle Imaging,” which is hereby incorporated by reference in itsentirety and for all purposes.

TECHNICAL FIELD

This disclosure relates generally to image or video capture devices,including a multiple camera system for generating an image with a large(e.g., wide) field of view.

BACKGROUND

Many devices include one or more cameras. For example, a smartphone ortablet includes a front facing camera to capture selfie images and arear facing camera to capture an image of a scene (such as a landscapeor other scenes of interest to a device user). A user may wish tocapture an image of a scene that does not fit within a field of view ofa camera. Some devices include multiple cameras with different fields ofview based on a curvature of a camera lens directing light to the imagesensor. The user may thus use the camera with the desired field of viewof the scene based on the camera lens curvature to capture an image.

SUMMARY

Systems and techniques are described for digital imaging to generate animage with a large field of view. For example, a device can include afirst camera with a first image sensor that captures a first image basedon first light redirected by a first light redirection element. Thefirst light redirection element can redirect the first light from afirst path to a redirected first path toward the first camera. Thedevice can include a second camera with a second image sensor thatcaptures a second image based on second light redirected by a secondlight redirection element. The second light redirection element canredirect the second light from a second path to a redirected second pathtoward the second camera. The first camera, second camera, first lightredirection element, and second light redirection element can bearranged so that a virtual extension of the first path beyond the firstlight redirection element intersects with a virtual extension of thesecond path intersect beyond the second light redirection element. Theseelements can be arranged so that first lens of the first camera and asecond lens of the second camera virtually overlap based on the lightredirection without physically overlapping. The device can modify thefirst image and/or the second image using a perspective distortioncorrection, for instance to make the first image and the second imageappear to view the photographed scene from the same angle. The devicecan generate a combined image from the first image and the second image,for example by aligning and stitching the first image and the secondimage together. The combined image can have a larger field of view thanthe first image, the second image, or both.

In one example, an apparatus for digital imaging is provided. Theapparatus includes a memory and one or more processors coupled to thememory, the one or more processors (e.g., implemented in circuitry) andcoupled to the memory. The one or more processors are configured to andcan: receive a first image of a scene captured by a first image sensor,wherein a first light redirection element redirects a first light from afirst path to a redirected first path toward the first image sensor,wherein the first image sensor captures the first image based on receiptof the first light at the first image sensor; receive a second image ofthe scene captured by a second image sensor, wherein a second lightredirection element redirects a second light from a second path to aredirected second path toward the second image sensor, wherein thesecond image sensor captures the second image based on receipt of thesecond light at the second image sensor, wherein a virtual extension ofthe first path beyond the first light redirection element intersectswith a virtual extension of the second path intersect beyond the secondlight redirection element; modify at least one of the first image andthe second image using a perspective distortion correction; and generatea combined image from the first image and the second image in responseto modification of the at least one of the first image and the secondimage using the perspective distortion correction, wherein the combinedimage includes a combined image field of view that is larger than atleast one of a first field of view of the first image and a second fieldof view of the second image.

In another example, a method of digital imaging is provided. The methodincludes receiving a first image of a scene captured by a first imagesensor, wherein a first light redirection element redirects a firstlight from a first path to a redirected first path toward the firstimage sensor, wherein the first image sensor captures the first imagebased on receipt of the first light at the first image sensor. Themethod includes receiving a second image of the scene captured by asecond image sensor, wherein a second light redirection elementredirects a second light from a second path to a redirected second pathtoward the second image sensor, wherein the second image sensor capturesthe second image based on receipt of the second light at the secondimage sensor, wherein a virtual extension of the first path beyond thefirst light redirection element intersects with a virtual extension ofthe second path intersect beyond the second light redirection element.The method includes modifying at least one of the first image and thesecond image using a perspective distortion correction. The methodincludes generating a combined image from the first image and the secondimage in response to modification of the at least one of the first imageand the second image using the perspective distortion correction,wherein the combined image includes a combined image field of view thatis larger than at least one of a first field of view of the first imageand a second field of view of the second image.

In another example, a non-transitory computer readable storage medium isprovided that has stored thereon instructions that, when executed by oneor more processors, cause the one or more processors to: receive a firstimage of a scene captured by a first image sensor, wherein a first lightredirection element redirects a first light from a first path to aredirected first path toward the first image sensor, wherein the firstimage sensor captures the first image based on receipt of the firstlight at the first image sensor; receive a second image of the scenecaptured by a second image sensor, wherein a second light redirectionelement redirects a second light from a second path to a redirectedsecond path toward the second image sensor, wherein the second imagesensor captures the second image based on receipt of the second light atthe second image sensor, wherein a virtual extension of the first pathbeyond the first light redirection element intersects with a virtualextension of the second path intersect beyond the second lightredirection element; modify at least one of the first image and thesecond image using a perspective distortion correction; and generate acombined image from the first image and the second image in response tomodification of the at least one of the first image and the second imageusing the perspective distortion correction, wherein the combined imageincludes a combined image field of view that is larger than at least oneof a first field of view of the first image and a second field of viewof the second image.

In another example, an apparatus for digital imaging is provided. Theapparatus includes means for receiving a first image of a scene capturedby a first image sensor, wherein a first light redirection elementredirects a first light from a first path to a redirected first pathtoward the first image sensor, wherein the first image sensor capturesthe first image based on receipt of the first light at the first imagesensor. The apparatus includes means for receiving a second image of thescene captured by a second image sensor, wherein a second lightredirection element redirects a second light from a second path to aredirected second path toward the second image sensor, wherein thesecond image sensor captures the second image based on receipt of thesecond light at the second image sensor, wherein a virtual extension ofthe first path beyond the first light redirection element intersectswith a virtual extension of the second path intersect beyond the secondlight redirection element. The apparatus includes means for modifying atleast one of the first image and the second image using a perspectivedistortion correction. The apparatus includes means for generating acombined image from the first image and the second image in response tomodification of the at least one of the first image and the second imageusing the perspective distortion correction, wherein the combined imageincludes a combined image field of view that is larger than at least oneof a first field of view of the first image and a second field of viewof the second image.

In some aspects, modifying at least one of the first image and thesecond image using the perspective distortion correction includes:modifying the first image from depicting a first perspective todepicting a common perspective using the perspective distortioncorrection; and modifying the second image from depicting a secondperspective to depicting the common perspective using the perspectivedistortion correction, wherein the common perspective is between thefirst perspective and the second perspective.

In some aspects, modifying at least one of the first image and thesecond image using the perspective distortion correction includes:identifying depictions of one or more objects in image data of at leastone of the first image and the second image; and modifying the imagedata by projecting the image data based on the depictions of the one ormore objects.

In some aspects, generating the combined image from the first image andthe second image, the one or more processors includes: aligning a firstportion of the first image with a second portion of the second image;and stitching the first image and the second image together based on thefirst portion of the first image and the second portion of the secondimage being aligned.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: modifying at least one of the firstimage and the second image using a brightness uniformity correction.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: the first image sensor; the secondimage sensor; the first light redirection element; and the second lightredirection element.

In some aspects, the first light redirection element includes a firstreflective surface, wherein, to redirect the first light toward thefirst image sensor, the first light redirection element uses the firstreflective surface to reflect the first light toward the first imagesensor; and the second light redirection element includes a secondreflective surface, wherein, to redirect the second light toward thesecond image sensor, second light redirection element uses the secondreflective surface to reflect the second light toward the second imagesensor.

In some aspects, the first light redirection element includes a firstprism configured to refract the first light; and the second lightredirection element includes a second prism configured to refract thesecond light. In some aspects, the first prism and the second prism arecontiguous. In some aspects, a bridge joins a first edge of the firstprism and a second edge of the second prism, wherein the bridge isconfigured to prevent reflection of light from at least one of firstedge of the first prism and the second edge of the second prism. In someaspects, the first prism includes at least one chamfered edge, andwherein the second prism includes at least one chamfered edge. In someaspects, the first prism includes at least one edge with alight-absorbing coating, wherein the second prism includes at least oneedge with the light-absorbing coating. In some aspects, the first pathis a path of the first light before the first light enters the firstprism, wherein the second path is a path of the second light before thesecond light enters the second prism. In some aspects, the first prismincludes a first reflective surface configured to reflect the firstlight, wherein the second prism includes a second reflective surfaceconfigured to reflect the second light. In some aspects, the first pathis a path of the first light after the first light enters the firstprism but before the first reflective surface reflects the first light,wherein the second path is a path of the second light after the secondlight enters the second prism but before the second reflective surfacereflects the second light.

In some aspects, the first image and the second image are capturedcontemporaneously. In some aspects, the first light redirection elementis fixed relative to the first image sensor, wherein the second lightredirection element is fixed relative to the second image sensor. Insome aspects, a first planar surface of the first image sensor faces afirst direction, wherein a second planar surface of the second imagesensor faces a second direction that is parallel to the first direction.

In some aspects, the apparatus comprises a camera, a mobile handset, asmart phone, a mobile telephone, a portable gaming device, anothermobile device, a wireless communication device, a smart watch, awearable device, a head-mounted display (HMD), an extended realitydevice (e.g., a virtual reality (VR) device, an augmented reality (AR)device, or a mixed reality (MR) device), a personal computer, a laptopcomputer, a server computer, another device, or a combination thereof.In some aspects, the one or more processors include an image signalprocessor (ISP). In some aspects, the apparatus includes a camera ormultiple cameras for capturing one or more images. In some aspects, theapparatus includes an image sensor that captures the image data. In someaspects, the apparatus further includes a display for displaying theimage, one or more notifications associated with processing of theimage, and/or other displayable data.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are illustrated by way of example, andnot by way of limitation, in the figures of the accompanying drawingsand in which like reference numerals refer to similar elements.Illustrative embodiments of the present application are described indetail below with reference to the following figures:

FIG. 1 is a conceptual diagram illustrating an example of a distortionin an image captured using a camera with a lens having lens curvature;

FIG. 2 is a conceptual diagram illustrating an example wide angle imagecapture based on a sequence of captures by a camera;

FIG. 3 is a conceptual diagram illustrating an example ghostingdistortion in a wide angle image generated using panoramic stitching;

FIG. 4 is a conceptual diagram illustrating an example stitchingdistortion in a wide angle image generated using panoramic stitching;

FIG. 5 is a block diagram illustrating an example device configured togenerate one or more wide angle images;

FIG. 6 is a conceptual diagram illustrating two image sensors and theirassociated lenses of two cameras for capturing image frames;

FIG. 7 is a conceptual diagram illustrating an example redirectionelement redirecting light to a camera lens and the change in position ofthe camera lens and associated image sensor based on the redirectionelement;

FIG. 8 is a conceptual diagram illustrating an example configuration oftwo cameras to generate a wide angle image using redirection elementsincluding mirrors;

FIG. 9 is a conceptual diagram illustrating an example configuration oftwo cameras to generate a wide angle image using redirection elementsincluding prisms;

FIG. 10A is a conceptual diagram illustrating an example perspectivedistortion in an image frame captured by one or more of the cameras;

FIG. 10B is a conceptual diagram illustrating an example perspectivedistortion correction of two image frames to a common perspective;

FIG. 10C is a conceptual diagram illustrating an example digitalalignment and stitching of two image frames captured by two cameras togenerate a wide angle image;

FIG. 10D is a conceptual diagram illustrating an example brightnessuniformity correction of a wide angle image generated from two imageframes captured by two cameras;

FIG. 11 is a conceptual diagram illustrating example light reflectionsfrom a camera lens that may cause scattering noise in a portion of animage frame;

FIG. 12A is a conceptual diagram illustrating an example redirectionelement to redirect light to a first camera and to redirect light to asecond camera;

FIG. 12B is a conceptual diagram illustrating the redirection element inFIG. 12A that illustrates the elimination of light scattering from aprism edge;

FIG. 12C is a conceptual diagram illustrating the redirection element inFIG. 12A from a perspective view;

FIG. 13A is a flow diagram illustrating an example process forgenerating a combined image from multiple image frames;

FIG. 13B is a flow diagram illustrating an example method of digitalimaging;

FIG. 14 is a flow diagram illustrating an example process for capturingmultiple image frames to be combined to generate a combined image frame;

FIG. 15 is a conceptual diagram illustrating examples of a flatperspective distortion correction and a curved perspective distortioncorrection;

FIG. 16 is a conceptual diagram illustrating pixel mapping from an imagesensor image plane to a perspective-corrected image plane in a flatperspective distortion correction and in a curved perspective distortioncorrection;

FIG. 17 is a conceptual diagram illustrating three example combinedimages of a scene that each have different degrees of curvature ofcurved perspective distortion correction applied;

FIG. 18 is a conceptual diagram illustrating a graph comparing differentdegrees of curvature of curved perspective distortion correction withrespect to a flat perspective distortion;

FIG. 19 is a flow diagram illustrating an example process for performingcurved perspective distortion correction;

FIG. 20 is a block diagram illustrating an example of an architecture ofan image capture and processing device; and

FIG. 21 is a block diagram illustrating an example of a system forimplementing certain aspects of the present technology.

DETAILED DESCRIPTION

Aspects of the present disclosure may be used for image or video capturedevices. Some aspects include generating a wide angle image usingmultiple cameras.

A smartphone, tablet, digital camera, or other device includes a camerato capture images or video of a scene. The camera has a maximum field ofview based on an image sensor and one or more camera lenses. Forexample, a single lens or multiple lens system with more curvature inthe camera lenses may allow a larger field of view of a scene to becaptured by an image sensor. Some devices include multiple cameras withdifferent fields of view based on curvatures of the focus lenses. Forinstance, a device may include a camera with a normal lens having anormal field of view, and a different camera with a wide-angle lenshaving a wider field of view. A user of the camera, or softwareapplication running on the camera's processor, can select between thedifferent cameras based on field of view, to select the camera with afield of view that is optimal for capturing a particular set of imagesor video. For example, some smartphones include a telephoto camera, awide angle camera, and an ultra-wide angle camera with different fieldsof view. Before capture, the user or software application may selectwhich camera to use based on the field of view of each camera.Compensation for such distortion can be computationally expensive andinaccurate due to reliance on approximations. Applying distortioncompensation can retain some of the original distortion, canovercompensate, and/or can introduce other image artifacts.

However, the ultra-wide angle camera may have a field of view that isless than a desired field of view of the scene to be captured. Forexample, many users want to capture images or video with a field of viewof a scene larger than the field of view of the camera. A devicemanufacturer may increase the curvature of a camera lens to increase thefield of view of the camera. However, the device manufacturer may alsoneed to increase the size and complexity of the image sensor toaccommodate the larger field of view.

Additionally, lens curvature introduces distortion into the capturedimage frames from the camera. For instance, lens curvature can introduceradial distortion, such as barrel distortion, pincushion distortion, ormustache distortion. Digital image manipulation can, in some cases, beused to perform software-based compensation for radial distortion bywarping the distorted image with a reverse distortion. However,software-based compensation for radial distortion can be difficult andcomputationally expensive to perform. Moreover, software-basedcompensation generally relies on approximations and models that may notbe applicable in all cases, and can end up warping the imageinaccurately or incompletely. The resulting image with the compensationapplied may still retain some radial distortion, may end up distorted inan opposite manner to the original image due to overcompensation, or mayinclude other visual artifacts.

Systems and techniques are described for digital imaging to generate animage with a large field of view. A device can include a first camerathat captures a first image based on first light redirected by a firstlight redirection element and a second camera that captures a secondimage based on second light redirected by a second light redirectionelement. The first camera, second camera, first light redirectionelement, and second light redirection element can be arranged so that afirst lens of the first camera and a second lens of the second cameravirtually overlap based on the light redirection without physicallyoverlapping. For example, a first center of a first entrance pupil ofthe first lens of the first camera and a second center of a secondentrance pupil of a second lens of the second camera can virtuallyoverlap without physically overlapping. The device can generate acombined image from the first image and the second image, for example byaligning and stitching the first image and the second image together.The combined image can have a wider field of view than the first image,the second image, or both.

The device does not rely on wide-angle lenses with increased lenscurvature to generate its combined image having the large field of view.As a result, the cameras in the device can use lenses that do notintroduce the radial distortion that wide-angle lenses andultra-wide-angle lenses introduce, in which case there is little or noneed to apply radial distortion compensation. Thus, generation of thecombined image having the large field of view with the device can beboth less computationally expensive and more accurate than producing acomparable image with a camera having a curved lens that introducesradial distortion and a processor that then compensates for that radialdistortion. The individual cameras in the device can also each have asmaller and less complex image sensor than the image sensor in a camerawith a curved lens that introduces radial distortion. Thus, theindividual cameras in the device can draw less power, and require lessprocessing power to process, than the camera with the curved lens thatintroduces radial distortion.

FIG. 1 is a conceptual diagram 100 illustrating an example of adistortion in an image captured using a camera 112 with a lens 104having lens curvature. The distortion is based on the curvature of alens 104. The camera 112 includes at least the lens 104 and the imagesensor 106. The lens 104 directs light from the scene 102 to the imagesensor 106. The image sensor 106 captures one or more image frames.Captured image frame 108 is an example image frame that depicts thescene 102 and that is captured by the image sensor 106 of the camera112. The captured image frame 108 includes a barrel distortion, which isa type of radial distortion. The barrel distortion in the captured imageframe 108 causes the center of the scene 102 to appear stretched in thecaptured image frame 108 with reference to the edges of the scene, whilethe corners of the scene 102 appear to be pinched toward the center inthe captured image frame 108.

A device, such as the camera 112 or another image processing device, mayprocess the captured image frame 108 using distortion compensation toreduce the barrel distortion. However, the processing may create its owndistortion effects on the captured image frame 108. For example, thecenter of the scene 102 in the captured frame 108 may be normalized orotherwise adjusted with reference to the edges of the scene in thecaptured image frame 108. Adjusting the center may include stretchingthe corners of the scene in the captured image frame 108 to more closelyresemble a rectangle (or the shape of the image sensor if different thana rectangle). An example processed image frame 110 generated byprocessing the captured image frame 108 using distortion compensation isillustrated in FIG. 1. The example processed image frame 110 illustratesan example in which the distortion compensation overcompensates for thebarrel distortion and introduces a pincushion distortion, which isanother type of radial distortion. Stretching the corners too much whileprocessing the captured image frame 108 may introduce the pincushiondistortion for instance. Processing an image using distortioncompensation can also introduce other image artifacts.

The lens curvature of a lens 104 can be increased in order to increasethe field of view for captured image frames by the image sensor 106. Forexample, wide-angle lenses, ultra-wide-angle lenses, and fisheye lensesall typically exhibit high levels of lens curvature that generallyresult in barrel distortion, other types of radial distortion, or othertypes of distortion. As a result, the distortion increases in eachcaptured image frame 108 captured using such a lens, as in the barreldistortion illustrated in FIG. 1. The likelihood of distortioncompensation to introduce distortions or other image artifacts into aprocessed image frame 110, such as the pincushion distortion illustratedin FIG. 1, also increases with increased curvature in the lens 104.Therefore, images captured and/or generated using a lens 104 with anincreased lens curvature, including images with smaller fields of viewthan desired (e.g., a cropped image) are generally distorted or includeartifacts.

Some devices also include a software function to generate images with awider field of view using a single camera based on motion of the camera.For example, some camera applications include a camera-movementpanoramic stitching mode to generate images with wider fields of viewthan the camera. For a camera-movement panoramic stitching mode, a usermoves a camera while the camera captures a sequence of image framesuntil all of a scene is included in at least one of the image frames.The image frames are then stitched together to generate the wide angleimage.

FIG. 2 is conceptual diagram 200 illustrating an example wide angleimage capture of a scene 202 based on a sequence of captures by a camera206. The user 204 wishes to capture an image of the scene 202, but thefield of view required to depict the entire scene 202 is greater thanthe field of view of the camera 206. Therefore, the user 204 places thecamera 206 in a camera-movement panoramic stitching mode. The user 204positions the camera 206 in a first position indicated by a firstillustration of the camera 206 using dotted lines so that the field ofview of the camera is directed towards scene portion 210. The user 204instructs the camera 206 to begin image frame capture (such as bypressing a shutter button), and the camera 206 captures a first imageframe with the scene portion 210. The user 204 moves the camera 206(such as along the camera movement arc 208) to move the camera's fieldof view of the scene 102 along direction 216. After capturing the firstimage frame, the camera 206 captures a second image frame of the sceneportion 212 while the camera 206 is in a second position indicated by asecond illustration of the camera 206 using dotted lines. The secondposition of the camera 206 is located further along the direction 216than the first position of the camera 206. The second position of thecamera 206 is located further along the camera movement arc 208 than thefirst position of the camera 206. The user continues to move the camera206, and the camera 206 captures a third image frame of the sceneportion 214 while the camera 206 is in a third position indicated by anillustration of the camera 206 using solid lines. The third position ofthe camera 206 is located further along the direction 216 than thesecond position of the camera 206. The third position of the camera 206is located further along the camera movement arc 208 than the secondposition of the camera 206. After panning the camera 206 along thecamera movement arc 208 to capture image frames across the scene 202during image frame capture, the user 204 may stop the image framecaptures (such as by again pressing a shutter button or by letting go ofa shutter button that was continually held during image frame capture).After capture of the sequence of image frames, the camera 206 or anotherdevice may stitch the sequence of image frames together to generate acombined image of the scene 102 having a wider field of view than eachof the first image frame, the second image frame, and the third imageframe. For example, the first image frame of the scene portion 210, thesecond image frame of the scene portion 212, and the third image frameof the scene portion 214 (captured at different times) are stitchedtogether to generate the combined image depicting the entire scene 202,which can be referred to as a wide angle image of the entire scene 202.While three image frames are shown, a camera-movement panoramicstitching mode may be used to capture and combine two or more imageframes based on the desired field of view for the combined image.

For example, the camera 206 or another device can identify that a firstportion of the first image frame and a second portion of the secondimage frame both depict a shared portion of the scene 202. The sharedportion of the scene 202 is illustrated between two dashed verticallines that fall within both the first scene portion 210 and the secondscene portion 212. The camera 206 or other device can identify theshared portion of the scene 202 within the first image and the secondimage by detecting features of shared portion the scene 202 within boththe first image and the second image. The camera 206 or other device canalign the first portion of the first image with the second portion ofthe second image. The camera 206 or other device can generate a combinedimage from the first image and the second image by stitching the firstportion of the first image and the second portion of the second imagetogether. The camera 206 can similarly stitch together the second imageframe and the third image frame. For instance, the camera 206 or otherdevice can identify a second shared portion of the scene 202 depicted inthe third portion of the third image frame and a fourth portion of thesecond image frame. The camera 206 or other device can stitch togetherthe third portion of the third image frame and the fourth portion of thesecond image frame. Since a sequence of image frames are captured over aperiod of time while the camera 206 is moving along the camera movementarc 208, the camera-movement panoramic stitching mode illustrated inFIG. 2 may be limited to generating still images and not video, since asuccession of panoramic stitching combined images cannot be generatedquickly enough to depict fluid movement. Additionally, the camera 206being moved and the time lapse in capturing the sequence of image framescan introduce one or more distortions or artifacts into a generatedimage. Example distortions include ghosting distortions and stitchingdistortions. A ghosting distortion is an effect where multiple instancesof a single object may appear in a final image. A ghosting distortionmay be a result of local motion in the scene 202 during the sequence ofimage frame captures. An example of a ghosting distortion is illustratedin FIG. 3. A stitching distortion is an effect where edges may be brokenor objects may be split, warped, overlaid, and so on where two imageframes are stitched together. An example of a stitching distortion isillustrated in FIG. 4.

Distortions are also introduced by an entrance pupil of the camerachanging depths from the scene when the camera is moved. In other words,moving the camera changes a position of a camera's entrance pupil withreference to the scene. An entrance pupil associated with an imagesensor is the image of an aperture from a front of a camera (such asthrough one or more lenses preceding or located at the aperture to focuslight towards the image sensor).

For the depths of objects in a scene to not change with reference to amoving camera between image captures, the camera needs to be rotated atan axis centered at the entrance pupil of the camera. However, when aperson moves the camera, the person does not rotate the camera on anaxis at the center of the entrance pupil. For example, the camera may bemoved around an axis at the torso of the person moving the camera (orthe rotation also includes translational motion). Since the camerarotation is not on an axis at the entrance pupil, the position of theentrance pupil changes between image frame captures, and the imageframes are captured at different depths. A stitching distortion may be aresult of parallax artifacts caused by stitching together image framescaptured at different depths. A stitching distortion may also be aresult of global motion (which also includes a change in perspective ofthe camera when capturing the sequence of image frames).

Distortions and artifacts can also be introduced into the combined imagebased on varying speeds of the user's movement of the camera 206 alongthe camera movement arc 208. For example, certain image frames mayinclude motion blur in certain frames if motion of the camera 206 isfast. Likewise, if motion of the camera 206 is fast, the shared portionof the scene depicted in two consecutive image frames may be very small,potentially introducing distortions due to poor stitching. Distortionsand artifacts can also be introduced into the combined image if certaincamera settings of the camera 206, such as focus or gain, change betweenimage frame captures during the camera movement arc 208. Such changes incamera settings can produce visible seams between images in theresulting combined image.

The figures illustrated herein depict each lens of each camera at alocation of an entrance pupil for the camera. For example, this is thecase in FIGS. 6-9, FIG. 11, and FIGS. 12A-12C. While a camera lens isillustrated as a single camera lens in the figures to preventobfuscating aspects of the disclosure, the camera lens may represent asingle element lens or a multiple element lens system of a camera. Inaddition, the camera may have a fixed focus, or the camera may beconfigured for autofocus (for which one or more camera lenses may movewith reference to an image sensor). The present disclosure is notlimited to a specific example of an entrance pupil or its location, or aspecific example of a camera lens or its location depicted in thefigures.

FIG. 3 is a conceptual diagram 300 illustrating an example ghostingdistortion 310 in a wide angle image generated using panoramicstitching. Panoramic stitching can refer to the camera-movementpanoramic stitching mode of operation in FIG. 2. A device, in acamera-movement panoramic stitching mode, is to generate an image 308 ofthe scene 302. The user positions the device so that the device's cameracaptures a first image frame including a first scene portion 304 at afirst time. The user moves the device so that the device's cameracaptures a second image frame including the second scene portion 306 ata second time. The scene 302 includes a car moving from left to right inthe scene 302. As a result of the car moving in scene 302, the firstimage frame includes a substantial portion of the car also included inthe second image frame. When the two image frames are stitched together,the car may appear as multiple cars or portions of cars (illustrated asghosting distortion 310) in the resulting image 308.

On the other hand, if the car in the scene 302 is moving from right toleft instead of left to right, then the car may be at least partiallyomitted from the image 308 despite being present in the scene 302 duringcapture of the first image frame and/or during capture of the secondimage frame. For example, if the car is at least partially in the secondscene portion 306 at the first time during capture of the first imageframe, then the car may be at least partially omitted from the firstimage frame. If the car is at least partially in the first scene portion304 at the second time during capture of the second image frame, thenthe car may be at least partially omitted from the second image frame.The combined image 308 may thus at least partially omit the car, and insome cases may include more than one copy of a partially omitted car.This type of omission represents another type of distortion or imageartifact that can result from camera-movement panoramic stitchingthrough motion of a camera 206 as illustrated in FIG. 2.

FIG. 4 is a conceptual diagram 400 illustrating an example stitchingdistortion 410 in a wide angle image generated using panoramicstitching. Panoramic stitching can refer to the camera-movementpanoramic stitching mode of operation in FIG. 2. FIG. 4 further depictsa parallax artifact induced stitching distortion. A device, in thecamera-movement panoramic stitching mode, can generate a combined image408 of the scene 402. The user positions the device so that the device'scamera 206 captures a first image frame including a first scene portion404 at a first time. The user moves the device so that the device'scamera 206 captures a second image frame including a second sceneportion 406 at a second time. As a result of the camera 206 movingbetween image frame captures (with the position of the entrance pupilchanging) and/or the change in perspective of the first image frame andthe second image frame of the scene 402, there may exist parallax basedand camera movement based artifacts or distortions when the two imageframes are stitched together. For example, the combined image 408 isgenerated by stitching the first image frame and the second image frametogether. As shown, a stitch distortion 410 exists where a left portionof the tree does not align with a right portion of the tree, and where aleft portion of the ground does not align with a right portion of theground. While the example stitching distortion 410 is illustrated as alateral displacement between the portions of the scene captured in thetwo image frames, the stitching distortion 410 may also include arotational displacement or warping caused by attempts to align the imageframes during stitching. In this manner, lines that should be straightand uninterrupted in the scene may appear to break at an angle in afinal image, lines that should be straight may appear curved near astitch, lines that should be straight may suddenly change direction neara stitch, or objects may otherwise appear warped or distorted on oneside of the stitch compared to the other side as a result of a rotation.Distortions from stitching are enhanced by the movement of the singlecamera to capture the image frames over time. For example, in somecases, stitching distortions may cause an object in the scene to appearstretched, squished, slanted, skewed, warped, distorted, or otherwiseinaccurate in the combined image 408.

Another example distortion is a perspective distortion. Referring backto FIG. 2, the perspective of the camera 206 is from the right of thescene portion 210, and the perspective of the camera 206 is from theleft of the scene portion 214. Therefore, horizontal edges (such as ahorizon) may appear slanted in one direction in the first image frame,and the same horizontal edges (such as the horizon) may appear slantedin the opposite direction in the third image frame. A final image fromthe image frames stitched together may connect the opposite slantededges via an arc. For example, a horizon in combined images generatedusing a camera-movement panoramic stitching mode can appear curvedrather than flat. Such curvature is an example of a perspectivedistortion. To exacerbate the perspective distortion, the perspectivevaries based on the camera movement, which can be inconsistent betweendifferent instances of generating a wide angle image throughcamera-movement panoramic stitching. As a result, the cameraperspectives during one sequence of captured image frames can differfrom the camera perspectives during other sequences of captured imageframes.

As described above, distortions caused by increasing a lens curvature toincrease a field of view reduces the quality of the resulting images,which negatively impacts the user experience. Furthermore, distortionscaused by capturing a sequence of image frames over time (in acamera-movement panoramic stitching mode) to generate a wide angle imagereduces the quality of the resulting images, which negatively impactsthe user experience. Additionally, a camera-movement panoramic stitchingmode that entails capture of a sequence of image frames while a usermanually moves the camera may prevent the camera from performing videocapture or may cause parallax artifacts that are difficult to removebecause of the camera movement. Therefore, there is a need for a meansfor generating a wide angle image with a large field of view (includinga sequence of wide angle images with large fields of view for video)that prevent or reduce the above described distortions.

In some examples of panoramic stitching, multiple cameras are used tocapture image frames, which can allow panoramic stitching to beperformed without camera movement. Image frames captured by thedifferent cameras can be stitched together to generate a combined imagewith a field of view greater than the field of view of any one camera ofthe multiple cameras. As used below, such a combined image (with a fieldof view greater than the field of view of any one camera of the multiplecameras) is referred to as a wide angle image. The multiple cameras maybe positioned so that the center of their entrance pupils overlap (suchas virtually overlap). In this manner, the multiple cameras or a deviceincluding the multiple cameras is not required to be moved (which maycause the position of one or more entrance pupils to change). As aresult, no distortions caused by a device movement is introduced intothe generated wide angle images. In some implementations, the multiplecameras are configured to capture image frames concurrently and/orcontemporaneously. As used herein, concurrent capture of image framesmay refer to contemporaneous capture of the image frames. As usedherein, concurrent and/or contemporaneous capture of image frames mayrefer to at least a portion of the exposure windows overlapping forcorresponding image frames captured by the multiple cameras. As usedherein, concurrent and/or contemporaneous capture of image frames mayrefer to at least a portion of the exposure windows for correspondingimage frames falling within a shared time window. The shared time windowmay, for example, have a duration of one or more picoseconds, one ormore nanoseconds, one or more milliseconds, one or more centiseconds,one or more deciseconds, one or more seconds, or a combination thereof.In this manner, no or fewer distortions caused by a time lapse incapturing a sequence of image frames is introduced into the generatedwide angle image.

In addition to overlapping the center of the entrance pupils, thecameras may be positioned with reference to each other to capture adesired field of view of a scene. Since the position of the cameras withreference to one another is known, a device may be configured to reduceor remove perspective distortions based on the known positioning.Additionally, because of images captured by multiple cameras captureconcurrently and/or contemporaneously does not require each camera tocapture a sequence of image frames as in the camera-movement panoramicstitching mode of FIG. 2, a device with multiple cameras may beconfigured to generate a wide angle video that includes a succession ofwide angle video frames. Each video frame can be a combined imagegenerated by stitching together two or more images from two or morecameras.

In the following description, numerous specific details are set forth,such as examples of specific components, circuits, and processes toprovide a thorough understanding of the present disclosure. The term“coupled” as used herein means connected directly to or connectedthrough one or more intervening components or circuits. Also, in thefollowing description and for purposes of explanation, specificnomenclature is set forth to provide a thorough understanding of thepresent disclosure. However, it will be apparent to one skilled in theart that these specific details may not be required to practice theteachings disclosed herein. In other instances, well known circuits anddevices are shown in block diagram form to avoid obscuring teachings ofthe present disclosure. Some portions of the detailed descriptions whichfollow are presented in terms of procedures, logic blocks, processingand other symbolic representations of operations on data bits within acomputer memory. In the present disclosure, a procedure, logic block,process, or the like, is conceived to be a self-consistent sequence ofsteps or instructions leading to a desired result. The steps are thoserequiring physical manipulations of physical quantities. Usually,although not necessarily, these quantities take the form of electricalor magnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a computer system.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present application,discussions utilizing the terms such as “accessing,” “receiving,”“sending,” “using,” “selecting,” “determining,” “normalizing,”“multiplying,” “averaging,” “monitoring,” “comparing,” “applying,”“updating,” “measuring,” “deriving,” “settling,” “generating” or thelike, refer to the actions and processes of a computer system, orsimilar electronic computing device, that manipulates and transformsdata represented as physical (electronic) quantities within the computersystem's registers and memories into other data similarly represented asphysical quantities within the computer system memories or registers orother such information storage, transmission or display devices.

In the figures, a single block may be described as performing a functionor functions; however, in actual practice, the function or functionsperformed by that block may be performed in a single component or acrossmultiple components, and/or may be performed using hardware, usingsoftware, or using a combination of hardware and software. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps aredescribed below generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present disclosure. Also, the example devices may includecomponents other than those shown, including well-known components suchas a processor, memory, and the like.

Aspects of the present disclosure are applicable to any suitableelectronic device including or coupled to multiple image sensors capableof capturing images or video (such as security systems, smartphones,tablets, laptop computers, digital video and/or still cameras, imagecapture devices 2005A, image processing devices 2005B, image capture andprocessing devices 2000, computing systems 2100, and so on). The terms“device” and “apparatus” are not limited to one or a specific number ofphysical objects (such as one smartphone, one camera controller, oneprocessing system and so on). As used herein, a device may be anyelectronic device with one or more parts that may implement at leastsome portions of the disclosure. While the below description andexamples use the term “device” to describe various aspects of thedisclosure, the term “device” is not limited to a specificconfiguration, type, or number of objects. As used herein, an apparatusmay include a device or a portion of the device for performing thedescribed operations.

Depictions in the figures may not be drawn to scale or proportion, andimplementations may vary in size or dimensions than as depicted in thefigures. Some of the figures depict a camera lens indicating an entrancepupil of a camera. However, the lenses and entrances pupils may be inany suitable positioning with reference to each other (and the imagesensors) to perform aspects of the present disclosure. A lens depictedin the figures may indicate a single element lens or a multiple elementlens (even though a lens may appear to be depicted as a single elementlens in the figures). Therefore, the present disclosure is not limitedto examples explicitly depicted in the figures.

FIG. 5 is a block diagram illustrating an example device 500 configuredto generate one or more wide angle images. The example device 500includes (or is coupled to) a camera 501 and a camera 502. While twocameras are depicted, the device 500 may include any number of cameras(such as 3 cameras, 4 cameras, and so on). The first camera 501 and thesecond camera 502 may be included in a single camera module or may bepart of separate camera modules for the device 500. In the example of asmartphone or tablet, the first camera 501 and the second camera 502 maybe associated with one or more apertures on a same side of the device toreceive light for capturing image frames of a scene. The first camera501 and the second camera 502 may be positioned with reference to oneanother to allow capture of a scene by combining images from the withcamera 501 and camera 502 to produce a field of view greater than thefield of view of the first camera 501 and/or the second camera 502. Insome implementations, the device 500 includes (or is coupled to) one ormore light redirection elements 503. At least a first subset of the oneor more light redirection elements 503 may redirect light towards thefirst camera 501. At least a second subset of the one or more lightredirection elements 503 may redirect light towards the second camera502. The first camera 501 can capture a first image based on incidentlight redirected by the one or more light redirection elements 503. Thesecond camera 502 can capture a second image based on incident lightredirected by the one or more light redirection elements 503. The device500 may combine the first image and the second image in order togenerate a combined image having a combined image field of view that iswider and/or larger than a first field of view of the first image, asecond field of view of the second image, or both. The combined imagemay be referred to as a wide angle image. The combined image field ofview may be referred to as a large field of view, a wide field of view,or a combination thereof.

The device 500 may generate the combined image by combining the firstimage and the second image, for instance by stitching together the firstimage and the second image without any need for movement of the firstcamera 501 and/or the second camera 502. For example, the device oranother device can identify that a first portion of the first imagecaptured by the first camera 501 and a second portion of the secondimage captured by the second camera 502 both depict a shared portion ofthe photographed scene. The device 500 can identify the shared portionof the scene within the first image and the second image by detectingfeatures of shared portion the scene within both the first image and thesecond image. The device 500 can align the first portion of the firstimage with the second portion of the second image. The device 500 cangenerate the combined image from the first image and the second image bystitching the first portion of the first image and the second portion ofthe second image together.

The first camera 501 and the second camera 502 may be proprietarycameras, specialized cameras, or any type of cameras. In some aspects,the first camera 501 and the second camera 502 may be the same type ofcamera as one another. For instance, the first camera 501 and the secondcamera 502 may be the same make and model. In some aspects, the firstcamera 501 and the second camera 502 may be different types, makes,and/or models of cameras. While the examples below depict two similarcameras 501 and 502, any suitable number, types, or configurations ofcameras may be used in performing aspects of the present disclosure. Thefirst camera 501 and the second camera 502 may each be configured toreceive and capture at least one spectrum of light, such as the visiblelight spectrum, the infrared light spectrum, the ultraviolet lightspectrum, the microwave spectrum, the radio wave spectrum, the x-rayspectrum, the gamma ray spectrum, another subset of the electromagneticspectrum, or a combination thereof.

The first camera 501, the second camera 502, and the one or moreredirection elements 503 may be arranged such that the center of theentrance pupils associated with the first camera 501 and the secondcamera 502 virtually overlap. For example, each camera includes an imagesensor coupled to one or more lenses to focus light onto thecorresponding image sensor, and a lens and entrance pupil are at thesame location for the camera. In using the one or more redirectionelements 503, the first camera 501 and the second camera 502 may bearranged such that their lenses virtually overlap (e.g., the centers oftheir respective entrance pupils virtually overlap) without their lensesphysically overlapping or otherwise occupying the same space. Forexample, light to be captured by the first camera 501 and the secondcamera 502 may be redirected (e.g., reflected and/or refracted) by theone or more redirection elements 503 so that the lenses of the firstcamera 501 and the second camera 502 can be physically separate whilemaintaining a virtual overlap of the lenses (e.g., a virtual overlap ofthe centers of the entrance pupils of the cameras). A parallax effectbetween image frames captured by the different camera 501 and 502 isreduced (or eliminated) as a result of the cameras' associated centersof the entrance pupils virtually overlapping.

As used herein, a virtual overlap may refer to a location that wouldinclude multiple objects (such as camera lenses) if the light is notredirected (such as described with reference to FIG. 7). For example,first lens of the first camera 501 and the second lens of the secondcamera 502 virtually overlapping can include a first virtual position ofthe first lens overlapping with a second virtual position of the secondlens. A first light travels along a first path before the first lightredirection element of the light redirection elements 503 redirects thefirst light away from the first path and toward the first camera 501. Asecond light travels along a second path before a second lightredirection element of the light redirection elements 503 redirects thesecond light away from the second path and toward the second camera 502.A virtual extension of the first path beyond the first light redirectionelement intersects with the first virtual position of the first lens. Avirtual extension of the second path beyond the second light redirectionelement intersects with the second virtual position of the first lens.

The device 500 may also include one or more additional lenses, one ormore apertures, one or more shutters, or other suitable components thatare associated with the first camera 501 and the second camera 502. Thedevice 500 may also include a flash, a depth sensor, or any othersuitable imaging components. While two cameras are illustrated as partof the device 500, the device 500 may include or be coupled toadditional image sensors not shown. In this manner, wide angle imagingmay include the use of more than two cameras (such as three or morecameras). The two cameras are illustrated for the examples below forclarity in explaining aspects of the disclosure, but the disclosure isnot limited to the specific examples of using two cameras.

The example device 500 also includes a processor 504, a memory 506storing instructions 508, and a camera controller 510. In someimplementations, the device 500 may include a display 514, a number ofinput/output (I/O) components 516, and a power supply 518. The device500 may also include additional features or components not shown. In oneexample, a wireless interface, which may include a number oftransceivers and a baseband processor, may be included for a wirelesscommunication device. In another example, one or more motion sensors(such as a gyroscope), position sensors (such as a global positioningsystem sensor (GPS)), and a sensor controller may be included in adevice.

The memory 506 may be a non-transient or non-transitory computerreadable medium storing computer-executable instructions 508 to performall or a portion of one or more operations described in this disclosure.In some implementations, the instructions 508 include instructions foroperating the device 500 in a wide angle capture mode using the firstcamera 501 and the second camera 502. The instructions 508 may alsoinclude other applications or programs executed by the device 500, suchas an operating system, a camera application, or other applications oroperations to be performed by the device 500. In some examples, thememory 506 stores image frames (as a frame buffer) for the first camera501 and/or for the second camera 502.

In some examples, the memory 506 stores camera brightness uniformitycalibration data. Using the camera brightness uniformity calibrationdata, the device 500 (e.g., the camera controller 510, the ISP 512,and/or the processor 504) can adjust brightness levels in a first imagefrom the first camera 501 and/or brightness levels in a second imagefrom the second camera 502. For instance, the device 500 can removevignetting or other brightness non-uniformities from the first image,the second image, or both. The device 500 can also increase or decreaseoverall brightness in the first image, the second image, or both, sothat overall brightness matches between the first image and secondimage. Such brightness adjustments can ensure that there is no visibleseam in the combined image (e.g., between the portion of the combinedimage that is from the first image and the portion of the combined imagethat is from the second image). In some examples, the memory 506 storesperspective distortion correction data. The perspective distortioncorrection data can include data such as angles, distances, directions,amplitudes, distortion correction vectors, curvatures, or a combinationthereof. Using the perspective distortion correction data, the device500 (e.g., the camera controller 510, the ISP 512, and/or the processor504) can perform perspective distortion correction (e.g., perspectivedistortion correction 1022, flat perspective distortion correction 1515,curved perspective distortion correction 1525, curved perspectivedistortion correction 1630).

The processor 504 may be one or more suitable processors capable ofexecuting scripts or instructions of one or more software programs (suchas instructions 508) stored within the memory 506. In some aspects, theprocessor 504 may be one or more general purpose processors that executeinstructions 508. For example, the processor 504 may be an applicationsprocessor and may execute a camera application. In some implementations,the processor 504 is configured to instruct the camera controller 510 toperform one or more operations with reference to the first camera 501and the second camera 502. In additional or alternative aspects, theprocessor 504 may include integrated circuits or other hardware toperform functions or operations without the use of software.

While shown to be coupled to each other via the processor 504 in theexample of FIG. 5, the processor 504, the memory 506, the cameracontroller 510, the optional display 514, and the optional I/Ocomponents 516 may be coupled to one another in various arrangements.For example, the processor 504, the memory 506, the camera controller510, the optional display 514, and/or the optional I/O components 516may be coupled to each other via one or more local buses (not shown forsimplicity).

If the device 500 includes a display 514, the display 514 may be anysuitable display or screen allowing for user interaction and/or topresent items for viewing by a user (such as captured images, video, orpreview images from one or more of the first camera 501 and the secondcamera 502). In some aspects, the display 514 is a touch-sensitivedisplay. The optional I/O components 516 may include any suitablemechanism, interface, or device to receive input (such as commands) fromthe user and to provide output to the user. For example, the I/Ocomponents 516 may include a graphical user interface (GUI), keyboard,mouse, microphone and speakers, a squeezable bezel, one or more buttons(such as a power button), a slider, or a switch.

The camera controller 510 may include an image signal processor 512,which may be one or more image signal processors to process capturedimage frames provided by the one or more cameras 501 and 502. In someexample implementations, the camera controller 510 (such as the imagesignal processor 512) may also control operation of the first camera 501and the second camera 502. For example, the camera controller 510 (suchas the image signal processor 512) may receive instructions from theprocessor 504 to perform wide angle imaging, and the camera controller510 may initialize the first camera 501 and the second camera 502 andinstruct the first camera 501 and the second camera 502 to capture oneor more image frames that the camera controller 510 and/or processor 504combine into a combined image using panoramic stitching for wide angleimaging. The camera controller 510 may control other aspects of thefirst camera 501 and the second camera 502, such as operations forperforming one or more of automatic white balance, automatic focus, orautomatic exposure operations.

In some aspects, the image signal processor 512 includes one or moreprocessors configured to execute instructions from a memory (such asinstructions 508 from the memory 506, instructions stored in a separatememory coupled to the image signal processor 512, or instructionsprovided by the processor 504). For example, the image signal processor512 may execute instructions to process image frames from the firstcamera 501 and the second camera 502 to generate a wide angle image. Inaddition or alternative to the image signal processor 512 including oneor more processors configured to execute software, the image signalprocessor 512 may include specific hardware to perform one or moreoperations described in the present disclosure. The image signalprocessor 512 alternatively or additionally may include a combination ofspecific hardware and the ability to execute software instructions.

While the image signal processor 512 is depicted as part of the cameracontroller 510, the image signal processor 512 may be separate from thecamera controller 510. For example, the camera controller 510 to controlthe first camera 501 and the second camera 502 may be included in theprocessor 504 (such as embodied in instructions 508 executed by theprocessor 504 or embodied in one or more integrated circuits of theprocessor 504). The image signal processor 512 may be part of the imageprocessing pipeline from an image sensor (for capturing image frames) tomemory (for storing the image frames) and separate from the processor504.

While the following examples for performing wide angle imaging or imagecapture are described with reference to the example device 500 in FIG.5, any suitable device or apparatus may be used. For example, the deviceperforming wide angle imaging may be a portion of the device 500 (suchas a system on chip or components of an imaging processing pipeline). Inanother example, the device 500 may include a different configuration ofcomponents or additional components than as depicted.

The device 500 is configured to generate one or more wide angle imagesusing the first camera 501 and the second camera 502. For example, thefirst camera 501 and the second camera 502 are configured to captureimage frames, and the device 500 (such as the image signal processor512) is configured to process the image frames to generate a wide angleimage. As used herein, a wide angle image refers to an image with awider field of view than the first camera 501 or the second camera 502.In processing the image frames, the device 500 combines the image framesto generate the wide angle image (which may also be referred to as acombined image). The first camera 501 and the second camera 502 may bepositioned so that the centers of the associated entrance pupilsvirtually overlap. In this manner, parallax effects may be reduced orremoved. Processing may also include reducing distortions in the imageframes for the combined image (such as reducing perspective distortionsbased on the difference in positions between the first camera 501 andthe second camera 502 and nonuniform brightness distortions caused by aconfiguration of one or more camera lenses focusing light onto the imagesensor of camera 501 or 502). In some implementations, the first camera501 and the second camera 502 may be configured to capture image framesconcurrently and/or contemporaneously. In this manner, distortionscaused by global motion or local motion may be reduced or removed. Asnoted above, image frames being captured concurrently and/orcontemporaneously may refer to at least a portion of the exposurewindows for the image frames overlapping. The exposure windows mayoverlap in any suitable manner. For example, start of frame (SOF) forthe image frames may be coordinated, end of frame (EOF) for the imageframes may be coordinated, or there exists a range of time during whichall of the image frames are in their exposure window. As used herein,concurrent and/or contemporaneous capture of image frames may refer toat least a portion of the exposure windows for corresponding imageframes falling within a shared time window. The shared time window may,for example, have a duration of one or more picoseconds, one or morenanoseconds, one or more milliseconds, one or more centiseconds, one ormore deciseconds, one or more seconds, or a combination thereof.

In some implementations, the first camera 501 and the second camera 502are configured to capture image frames to appear as if the image sensorsof the first camera 501 and the second camera 502 border one another. Insome implementations, a first camera 501 and a second camera 502 may beat an angle from one another to capture different portions of a scene.For example, if a smartphone is in a landscape mode, the first camera501 and the second camera 502 may be neighboring each other horizontallyand offset from each other by an angle. The first camera 501 may capturea right portion of the scene, and the second camera 502 may capture aleft portion of the scene.

In some examples, the first camera 501, the second camera 502, or bothare stationary. In some examples, the lens of the first camera 501, thelens of the second camera 502, or both are stationary. In some examples,the image sensor of the first camera 501, the image sensor of the secondcamera 502, or both are stationary. In some examples, each of the one ormore light redirection elements 503 is stationary.

FIG. 6 is a conceptual diagram 600 illustrating a first camera and asecond camera. The first camera includes a first image sensor 602 and anassociated first camera lens 606, which are illustrated using dashedlines in FIG. 6. The first camera lens 606 is located at the entrancepupil of the first camera. The second camera includes a second imagesensor 604 and an associated second camera lens 608, which areillustrated using solid lines in FIG. 6. The second camera lens 608 islocated at the entrance pupil of the second camera. As noted above,while a camera lens may be depicted as a single lens, the camera lensmay be a single element lens or a multiple element lens system.

The conceptual diagram 600 may be an example of a conceptualconfiguration of the first camera 501 and the second camera 502 of thedevice 500. The conceptual depiction of the overlapping lenses 606 and608 illustrates the entrance pupil of the first camera virtuallyoverlapping with the entrance pupil of the second camera. Theoverlapping entrance pupil centers reduce or remove a parallax for imageframes captured by the different image sensors 602 and 604.Corresponding image frames from the image sensors 602 and 604 may becombined to generate an image with a larger field of view than anindividual image frame. For example, the images may be stitchedtogether. As noted above, reducing or removing the parallax reduces thenumber and effect of artifacts or distortions that may exist in thecombined image.

In some implementations, the field of view of the first image sensor 602overlaps the field of view of the second image sensor 604. For example,a right edge of the first image sensor's field of view may overlap aleft edge of the second image sensor's field of view.

Since the first image sensor 602 may capture a right portion of thescene in the wide angle image and the second image sensor 604 maycapture a left portion of the scene in the wide angle image, theperspective of the wide angle image may be generated to be between theperspective of the first image sensor 602 and the perspective of thesecond image sensor 604. The image sensors 602 and 604 are not parallelto each other, and the image frames captured by the image sensors 602and 604 include perspective distortions with reference to each other. Togenerate the wide angle image with a perspective between the twoperspectives, the device 500 may perform perspective distortioncorrection on image frames from both image sensors 602 and 604 togenerate image frames with a desired perspective. In some otherimplementations, the device 500 may perform perspective distortioncorrection on image frames from one image sensor to generate imageframes with a similar perspective as the other image sensor. In thismanner, a wide angle image may have a perspective of one of the imagesensors.

In addition to reducing or removing parallax artifacts, the device 500may reduce a perspective distortion with more success using theconfiguration shown in the conceptual diagram 600 than using a singlecamera in a camera-movement panoramic stitching mode that relies on asingle camera that is physically moved (such as depicted in FIG. 2) orwith more curvature of a camera lens to increase the field of view.Since the cameras have fixed positions with reference to each other, theangle between the image sensors 602 and 604 is static. Using theconfiguration shown in FIG. 6, the device 500 may process the capturedimage frames to reduce perspective distortion based on the angle. Sincethe angle is static, the perspective distortion may be correcteddigitally (such as during processing of the captured image frames). Forexample, the device 500 may perform perspective distortion correction asa predefined filter (such as in the image signal processor 512) that isconfigured based on the angle between the image sensors 602 and 604. Incontrast, angles between instances of an image sensor (for acamera-movement panoramic stitching mode that relies on a single camerathat is physically moved as in FIG. 2) when being moved between imageframe captures may vary depending on the device movement. Therefore, adevice using a camera-movement panoramic stitching mode that relies on asingle camera that is physically moved (as in FIG. 2) cannot use apredefined filter based on a static angle to remove perspectivedistortion, since the static angle does not exist. This makesperspective distortion very difficult and computationally expensive tocompensate for in combined images generated using camera-movementpanoramic stitching that relies on a single camera that is physicallymoved as in FIG. 2. A device 500 with fixed positions for the firstcamera 501, the second camera 502, and/or the one or more lightredirection elements 503 can therefore perform perspective distortioncorrection more quickly, reliably, and at reduced computational expense.

Referring back to FIG. 6, the first camera and the second camera mayhave the same focal length. In this manner, the range of depths of thescene in focus is the same for the image sensors 602 and 604. However,the lenses 606 and 608 may not physically occupy the same space. In someimplementations, a prism and/or a reflective surface may be configuredto perform the functions of the spatially overlapped two lenses (withoutphysical contact between separate lenses). For example, a prism and/or areflective surface may be shaped to direct light from a first directionto the first camera lens 606 and direct light from a second direction tothe second camera lens 608 such that the virtual images of the entrancepupils associated with the camera lenses 606 and 608 overlap at theircenters.

In some other implementations, the cameras may be configured so that thecenter of the entrance pupils are virtually overlapping while the cameralenses of the cameras are spatially separated from one another. Forexample, one or more light redirection elements may be used to redirectlight towards the camera lenses 606 and 608. Based on the properties andposition of a light redirection element, the first camera lens 606 maybe spatially separated from the second cameras lens 608 while the centerof the entrance pupils virtually overlap. In this manner, the imagesensors may still be configured to capture image frames that conform tothe conceptual diagram 600 of having overlapping camera lens 606 and 608in FIG. 6. In some implementations, the first image sensor 602 may beassociated with a first redirection element, and the second image sensor604 may be associated with a second redirection element. In someimplementations, the first redirection element and the secondredirection element may be the same redirection element (e.g., as in theredirection element 1210 of FIGS. 12A-12C).

As used herein, a redirection element may be any suitable elementconfigured to redirect light traveling along a first path towards asecond path. The redirection element may reflect or refract the light.In some implementations, the redirection element may include a mirror toreflect the light. As used herein, a mirror may refer to any suitablereflective surface (such as a reflective coating, mirrored glass, and soon).

FIG. 7 is a conceptual diagram 700 illustrating a redirection element706 redirecting light to an image sensor 702 and the change in positionof the image sensor 702 based on the redirection element 706. Asdepicted, the redirection element 706 may include a mirror to reflectthe light received towards the lens 704 (and the image sensor 702). Thepath of the light is illustrated using solid lines with arrow indicatorsindicating direction of the light. If the redirection element 706 wereremoved, omitted, or otherwise did not exist, the light would insteadtravel to a location of the virtual image sensor 708 (via the virtualentrance pupil of the virtual camera lens 710) along an extension of thelight's original path (illustrated using dotted lines) before the lightwas redirected by the light redirection element 706. For example,referring back to FIG. 6, the light to be directed to the second imagesensor 604 approaches the location of the camera lens 608. Referring toFIG. 7, if a light redirection element 706 is used to direct light tothe image sensor 702 through the camera lens 704, the image sensor 702is positioned as depicted in FIG. 7 instead of at the position of thevirtual image sensor 708 for the image sensor 702 to capture the sameimage frame. In this manner, the location of the camera lens 704 is asdepicted in FIG. 7 instead of at the position of the virtual camera lens710. In this manner, the lenses for multiple image sensors may bespatially separated with the lenses and/or entrance pupils stillvirtually overlapping.

For example, a first ray of light follows an initial path 720 beforereaching the light redirection element 706 and being redirected onto aredirected path 722 directed to the camera lens 704 and the image sensor702. The first ray of light reaches the camera lens 704 and the imagesensor 702 along the redirected path 722. A virtual extension 724 of theinitial path 720 beyond the light redirection element 706 is illustratedin a dotted line and is instead directed to, and reaches, the virtualcamera lens 710 and the virtual image sensor 708. A second ray of lightand a third ray of light are also illustrated in FIG. 7. The lightredirection element 706 redirects the second ray of light and the thirdray of light from their initial paths toward the camera lens 704 and theimage sensor 702. The second ray of light and the third ray of lightthus reach the camera lens 704 and the image sensor 702. Virtualextensions of the initial paths of the second ray of light and the thirdray of light beyond the light redirection element 706 are illustratedusing dotted lines and are instead directed to, and reach, the virtualcamera lens 710 and the virtual image sensor 708.

The reflective surface (e.g., mirror) of the redirection element 706 canform a virtual image positioned behind the reflective surface (e.g.,mirror) of the redirection element 706 (to the right of the of theredirection element 706 as illustrated in FIG. 7). The virtual cameralens 710 may be a virtual image of the camera lens 704 observed throughthe reflective surface (e.g., mirror) of the redirection element 706from the direction of the initial path 720 are depicted in FIG. 7. Thevirtual image sensor 708 may be a virtual image of the image sensor 702observed through reflective surface (e.g., mirror) of the redirectionelement 706 from the direction of the initial path 720 are depicted inFIG. 7.

FIG. 8 is a conceptual diagram 800 illustrating an example configurationof two cameras to generate a wide angle image using redirection elements810 and 812. A first camera includes a first camera lens 806 (which maybe one or more camera lenses) and a first image sensor 802. A secondcamera includes a second camera lens 808 (which may be one or morecamera lenses) and a second image sensor 804.

The depiction 800 in FIG. 8 may achieve the same function as theconceptual diagram 600 in FIG. 6, where the first lens 806 and thesecond lens 808 virtually overlap (e.g., the center of the entrancepupils for the camera lenses 806 and 808 virtually overlap) while beingphysically spatially separated to remove or reduce parallax artifacts incombined images from image frames captured by the image sensors 802 and804. In comparing the depiction 800 to the conceptual diagram 600 inFIG. 6, the first image sensor 802 (associated with the firstredirection element 810) is configured to capture one portion of ascene, similar to the first image sensor 602. The second image sensor804 (associated with the second redirection element 812) is configuredto capture the other portion of the scene, similar to the second imagesensor 604. The first camera lens 806 is spatially separated from thesecond camera lens 808, and the first image sensor 802 is spatiallyseparated from the second image sensor 804 based on using the firstredirection element 810 and the second redirection element 812.

In some implementations, the redirection elements 810 and 812 may bepositioned on an outside of a device. For example, a component includingthe redirection elements may be coupled to the device 500 to directlight through one or more openings in the device 500 towards the imagesensors of the first camera 501 and the second camera 502. In someexamples, the device 500 may include the redirection elements disposedon an outer surface of the device 500. In some examples, the redirectionelements may be disposed inside of a device. For example, the device mayinclude one or more openings and/or apertures to allow light to enterthe device (such as light from the scene to be captured for generating awide angle image). The openings/apertures may include glass or anothertransparent material to allow light to pass, which may be shaped intoone or more lenses. The opening may or may not include one or morelenses or other components to adjust the direction of light into thedevice. The redirection elements 810 and 812 may be positioned along theoptical path between a device opening and the associated image sensor802 or 804.

While the redirection elements 810 and 812 are illustrated as twoseparate mirrors, the redirection elements 810 and 812 may be oneredirection element. For example, the redirection elements 810 and 812may physically connect on one side to be one redirection element.Additionally, the arrangement of the image sensors 802 and 804 areillustrated as being oriented towards each other. For instance, theoptical axes of the image sensors 802 and 804 may be aligned and/or maybe parallel to one another. However, the image sensors and lenses may bearranged in any suitable manner to receive light from a desired field ofview of a scene. For instance, the optical axes of the image sensors 802and 804 may be not aligned and/or may be not parallel to one anotherand/or may be at an angle relative to one another. The presentdisclosure is not limited to the arrangement of the components in thedepiction in FIG. 8.

In some implementations, the image sensors 802 and 804 are configured tocapture an image frame concurrently and/or contemporaneously (such as atleast a portion of the exposure windows overlapping for the imageframes). In this manner, local motion and global motion is reduced (thusreducing distortions in a generated wide angle image). In someimplementations, the image sensors 802 and 804 are configured to capturean image frame concurrently, contemporaneously, and/or within a sharedtime window. The shared time window may, for example, have a duration ofone or more picoseconds, one or more nanoseconds, one or moremilliseconds, one or more centiseconds, one or more deciseconds, one ormore seconds, or a combination thereof. Additionally, since the anglebetween the image sensors 802 and 804 is static, a device may beconfigured to reduce perspective distortion based on the known angles.

In some implementations, light to the first image sensor 802 and lightto the second image sensor 804 may be refracted (e.g., through a highrefractive index medium) to reduce a perspective distortion and/or lightvignetting at the camera aperture. Light propagating in a highrefractive index material has a smaller divergence angle before existingthe medium, reducing vignetting at a lens aperture that is located nearthe existing surface of the high refractive medium. Refraction mayalternatively or additionally be used to adjust a field of view of theimage sensors 802 and 804. For example, the field of view may be widenedto widen the field of view of the wide angle image. In another example,the field of view may be shifted to allow for different spacings betweenthe image sensors 802 and 804. Refraction may be used to allow furtherphysical separation between the camera lenses 806 and 808 while stillallowing the center of the entrance pupils to virtually overlap. Forexample, a prism may refract light intended for a respective imagesensor, and the prism may affect the location of the entrance pupilassociated with the image sensor. Based on the refraction, additionalphysical spacing between camera lenses may be allowed while stillallowing a virtual overlap of the center of the entrance pupils. In someimplementations, a redirection element may include a prism. At least oneof the surfaces on the prism can include a reflective surface, such as amirror. In this manner, one or more redirection elements includingprisms may be configured to refract and/or reflect light directedtowards the first image sensor 802 or the second image sensor 804.

FIG. 9 is a conceptual diagram 900 illustrating an example configurationof two cameras and two redirection elements 910 and 912. The two camerasare used to generate a wide angle image. A first camera includes a firstimage sensor 902 and a first camera lens 906. A second camera includes asecond image sensor 904 and a second camera lens 908.

The redirection elements 910 and 912 may include one or more prisms.Each prisms can include a high refractive index medium (e.g., having arefractive index above a threshold). As depicted, a first redirectionelement 910 redirects a first light (e.g., including one or more rays oflight) from a first path that approaches the first redirection element910 to a redirected first path towards the first image sensor 902. Thefirst path may be referred to as the initial first path. A secondredirection element 912 redirects a second light (e.g., including one ormore rays of light) from a second path that approaches the secondredirection element 912 to a redirected second path towards the secondimage sensor 904. The second path may be referred to as the initialsecond path. The location of the redirection elements 910 and 912 may beas described with reference to FIG. 8. For example, the redirectionelements 910 and 912 may be outside of the device, or the redirectionelements 910 and 912 may be inside of the device and configured toreceive light passing through an opening in the device.

In FIG. 9, the first lens 906 may also represent the position of anaperture of, and/or the entrance pupil for, the first camera. The secondlens 908 may also represent the position of an aperture of, and/or theentrance pupil for, the second camera. In the depiction 900, the firstredirection element 910 includes a first prism, and the secondredirection element 912 includes a second prism. The first prism isconfigured to refract the first light destined for the first imagesensor 902 to redirect the first light from a prism-approaching firstpath to a refracted first path. The second prism is configured torefract the second light destined for the second image sensor 904 toredirect the second light from a prism-approaching second path to arefracted second path. In some implementations, the first redirectionelement 910 also includes a first mirror on side 918 of the first prism.The first mirror is configured to reflect the first light towards thefirst image sensor 902 by redirecting the first light from the refractedfirst path to a reflected first path. The second redirection element 912also includes a second mirror on side 920 of the second prism. Thesecond mirror is configured to reflect the second light towards thesecond image sensor 904 by redirecting the second light from therefracted second path to a reflected second path. After being reflectedby the first mirror on side 918, the first light exits the first prism922.

Due to the refraction of the first prism 922, the first light may beredirected upon exiting the first prism 922, from the reflected firstpath to a post-prism first path. Similarly, after being reflected by thesecond mirror on side 920, the second light exits the second prism 922.Due to the refraction of the second prism 924, the second light may beredirected upon exiting the second prism 924, from the reflected secondpath to a post-prism second path.

In some examples, the first light may further be redirected (e.g., viarefraction) from the post-prism first path to a post-lens first path bythe first lens 906. In some examples, the second light may further beredirected (e.g., via refraction) from the post-prism second path to apost-lens second path by the second lens 908. In this manner, eachredirection element 910 and 912 may include a prism, with one side ofthe prism including a reflective coating. Light passing through theprism and reaching the reflective coating is reflected or folded backtowards the respective image sensor. In some other implementations, aredirection element may include separate reflective and refractivecomponents. For example, the first mirror or the second mirror may be aseparate component from the first prism and the second prism,respectively.

As used herein, a prism may refer to any suitable light refractingobject, such as a glass or plastic prism of a suitable shape. Suitableshapes may include a triangular prism, hexagonal prism, and so on withangles of surfaces configured to refract light from the scene asdesired. In some implementations, the redirection elements include anequilateral triangular prism (or other suitable sided triangular prismfor refracting light). In the depiction 900, side 922 of the firstredirection element 910 is approximately aligned on the same plane asside 924 of the second redirection element. The prisms may be configuredso that each camera includes an approximately 70 degree angle of view (afield of view having an angle of approximately 70 degrees). In someimplementations, the sides 922 and 924 are coated with ananti-reflective coating to prevent reflecting light to be captured bythe image sensor 902 and 904. In some implementations, the prismsurfaces that face the camera lenses are also coated with ananti-reflective coating to prevent light reflecting from these surfaces.

In some examples, the post-lens first path may be referred to as theredirected first path. In some examples, the post-prism first path maybe referred to as the redirected first path. In some examples, thereflected first path may be referred to as the redirected first path. Insome examples, the refracted first path may be referred to as theredirected first path. In some examples, the post-lens second path maybe referred to as the redirected second path. In some examples, thepost-prism second path may be referred to as the redirected second path.In some examples, the reflected second path may be referred to as theredirected second path. In some examples, the refracted second path maybe referred to as the redirected second path. In some examples, theprism-approaching first path may be referred to as the first path or asthe initial first path. In some examples, the refracted first path maybe referred to as the first path or as the initial first path. In someexamples, the prism-approaching second path may be referred to as thesecond path or as the initial second path. In some examples, therefracted second path may be referred to as the second path or as theinitial second path.

The first prism or the second prism may be configured to refract lightfrom a portion of the scene in order to adjust a focus distance. Forexample, the first prism and the second prism may be shaped such thatthe entrance and exit angles of light for the prisms allow theassociated camera lenses 906 and 908 to be in different positions whilestill having the same effect of the conceptual diagram 600 in FIG. 6. Inthis manner, the lenses may be spatially separated while the entrancepupils' centers still virtually overlap (as depicted in FIG. 6). Thevirtual overlap in the centers of the entrance pupils of the first lens906 and the second lens 908, illustrated as an actual overlap of theentrance pupils of the first virtual lens 926 and the second virtuallens 928, can provide the technical benefit of reducing or removingparallax artifacts in a combined image that might otherwise be present(and present a technical problem) if the entrance pupils did notvirtually overlap as they do in FIG. 9. For example, as a result of theredirection elements 910 and 912, the first image sensor 902 can beconceptualized as the first virtual image sensor 914 if the firstredirection element 910 does not exist, and the second image sensor 904can be conceptualized as the second virtual image sensor 914 if thesecond redirection element 912 does not exist. Similarly, lenses 906 and908 can be conceptualized as virtual lenses 926 and 928 if theredirection elements 910 and 912 do not exist. The overlapping virtuallenses 926 and 928 indicate overlapping entrance pupils, such asillustrated in FIG. 6.

The first virtual lens 926 can be conceptualized as a virtual position,orientation, and/or pose that the first lens 906 would have in order toreceive the first light that the first lens 906 actually receives, ifthat first light had continued along a virtual extension of its firstpath (extending beyond the first redirection element 910) instead ofbeing redirected toward the first lens 906 and the first image sensor902 by the at least part of the first redirection element 910. Thesecond virtual lens 928 can be conceptualized as a virtual position,orientation, and/or pose that the second lens 908 would have in order toreceive the second light that the second lens 908 actually receives, ifthat second light had continued along a virtual extension of its secondpath (extending beyond the second redirection element 912) instead ofbeing redirected toward the second lens 908 and the second image sensor904 by the at least part of the second redirection element 912.

Similarly, the first virtual image sensor 914 can be conceptualized as avirtual position, orientation, and/or pose that the first image sensor902 would have in order to receive the first light that the first imagesensor 902 actually receives, if that first light had continued along avirtual extension of its first path instead of being redirected towardthe first lens 906 and the first image sensor 902 by the at least partof the first redirection element 910. The second virtual image sensor916 can be conceptualized as a virtual position, orientation, and/orpose that the second image sensor 904 would have in order to receive thesecond light that the second image sensor 904 actually receives, if thatsecond light had continued along a virtual extension of its initialsecond path instead of being redirected toward the second lens 908 andthe second image sensor 904 by the at least part of the secondredirection element 912.

In some examples, the distance between the first redirection element 910and the first lens 906 is equal to the distance between the firstredirection element 910 and the first virtual lens 926. In someexamples, the distance between the first redirection element 910 and thefirst image sensor 902 is equal to the distance between the firstredirection element 910 and the first virtual image sensor 914. In someexamples, the distance between the second redirection element 912 andthe second lens 908 is equal to the distance between the secondredirection element 912 and the second virtual lens 928. In someexamples, the distance between the second redirection element 912 andthe second image sensor 904 is equal to the distance between the secondredirection element 912 and the second virtual image sensor 916.

In some examples, the optical distance between the reflection surface918 first redirection element 910 and the first lens 906 is about equalto the optical distance between the reflection surface of the firstredirection element 910 and the first virtual lens 926. In someexamples, the optical distance between the reflection surface of firstredirection element 910 and the first image sensor 902 is about equal tothe optical distance between the reflection surface of first redirectionelement 910 and the first virtual image sensor 914. In some examples,the optical distance between the reflection surface of the secondredirection element 912 and the second lens 908 is about equal to theoptical distance between the reflection surface of the secondredirection element 912 and the second virtual lens 928. In someexamples, the optical distance between the reflection surface of thesecond redirection element 912 and the second image sensor 904 is aboutequal to the optical distance between the second reflection surface ofthe redirection element 912 and the second virtual image sensor 916.

Identifying the virtual positions, orientations, and/or posescorresponding to the first virtual lens 926, the second virtual lens928, the first virtual image sensor 914, and the second virtual imagesensor 916 can include conceptual removal or omission of at least partof the first redirection element 910 and at least part the secondredirection element 912, such as conceptual removal or omission of atleast the reflective surface (e.g., mirror) on side 918 of the firstprism, the reflective surface (e.g., mirror) on side 920 of the secondprism, the first prism itself, the second prism itself, or a combinationthereof. The prior path of the first light can include the path of thefirst light before the first light entered the first prism or the pathof the first light after the first light entered the first prism butbefore the first light was redirected by the reflective surface (e.g.,mirror) on side 918 of the first prism. The prior path of the secondlight can include the path of the second light before the second lightentered the second prism or the path of the second light after thesecond light entered the second prism but before the second light wasredirected by the reflective surface (e.g., mirror) on side 920 of thesecond prism.

The first virtual lens 926 can be referred to as a virtual lens of thefirst lens 906, a virtual position of the first lens 906, a virtualorientation of the first lens 906, a virtual pose of the first lens 906,or a combination thereof. The second virtual lens 928 can be referred toas a virtual lens of the second lens 908, a virtual position of thesecond lens 908, a virtual orientation of the second lens 908, a virtualpose of the second lens 908, or a combination thereof. The first virtualimage sensor 914 can be referred to as a virtual image sensor of thefirst image sensor 902, a virtual position of the first image sensor902, a virtual orientation of the first image sensor 902, a virtual poseof the first image sensor 902, or a combination thereof. The secondvirtual image sensor 916 can be referred to as a virtual image sensor ofthe second image sensor 904, a virtual position of the second imagesensor 904, a virtual orientation of the second image sensor 904, avirtual pose of the second image sensor 904, or a combination thereof.Based on refraction, the spacing between the first camera lens 906 andthe second camera lens 908 may be less than the spacing between thefirst camera lens 806 and the second camera lens 808 in FIG. 8 (in whichthe light redirection elements may not refract light). Similarly, thespacing between the first image sensor 902 and the second image sensor904 may be less than the spacing between the first image sensor 802 andthe second image sensor 804 in FIG. 8.

The reflective surface (e.g., mirror) on side 918 of the firstredirection element 910 can form a virtual image positioned behind thereflective surface (e.g., mirror) on side 918 of the first redirectionelement 910 (below and to the right of the first redirection element 910as illustrated in FIG. 9). The reflective surface (e.g., mirror) on side920 of the second redirection element 912 can form a virtual imagepositioned behind the reflective surface (e.g., mirror) on side 920 ofthe second redirection element 912 (below and to the left of the secondredirection element 912 as illustrated in FIG. 9). The first virtuallens 926 may be a virtual image of the first lens 906 as observedthrough the reflective surface (e.g., mirror) on side 918 of the firstredirection element 910 from the direction of the light approaching thefirst redirection element 910 are depicted in FIG. 9. The first virtualimage sensor 914 may be a virtual image of the first image sensor 902 asobserved through the reflective surface (e.g., mirror) on side 918 ofthe first redirection element 910 from the direction of the lightapproaching the first redirection element 910 are depicted in FIG. 9.The second virtual lens 928 may be a virtual image of the second lens908 as observed through the reflective surface (e.g., mirror) on side920 of the second redirection element 912 from the direction of thelight approaching the second redirection element 912 are depicted inFIG. 9. The second virtual image sensor 916 may be a virtual image ofthe second image sensor 904 as observed through the reflective surface(e.g., mirror) on side 920 of the second redirection element 912 fromthe direction of the light approaching the second redirection element912 are depicted in FIG. 9.

In some implementations, the first prism and the second prism arephysically separated from each other (such as by ½ millimeter (mm)). Thespacing may be to prevent the prisms from bumping each other and causingdamage to the prisms. In some other implementations, the first prism andthe second prism may be physically connected. For example, the firstprism and the second prism may be connected at one of their corners sothat the first redirection element 910 and the second redirectionelement 912 are the same redirection element with multiple prisms andmirrors for refracting and reflecting light for the first image sensor902 and the second image sensor 904.

Similar to as described above with reference to FIG. 8, a perspectivedistortion may be reduced by performing a perspective distortioncorrection digitally to the image frames post-capture. The image frames(with the distortion corrected) may be combined (e.g., digitally) by adevice to generate a wide angle image (which may also be referred to asa combined image). Similar to FIG. 8, the image sensors 902 and 904 maybe configured to concurrently and/or contemporaneously capture imageframes, and/or to capture image frames within a shared time window, toreduce distortions from motion or other distortions in the combinedimage.

As noted above, image frames captured by the image sensors 802, 804,902, or 904 can include a perspective distortion. However, because theperspectives captured by the image sensors 802, 804, 902, and 904 areknown and are static, perspective distortion compensation techniques canin some cases be applied consistently to every image captured by each ofthe image sensors 802, 804, 902, and 904.

FIG. 10A is a conceptual diagram 1000 illustrating an exampleperspective distortion in an image frame 1006 captured by the imagesensor 1004. The image sensor 1004 may be an implementation of any ofthe image sensors in FIG. 8 or FIG. 9. As shown, the image sensor 1004captures the scene 1002 at an angle with reference to perpendicular tothe scene 1002. A lens (not pictured) may be positioned between thescene 1002 and the image sensor 1004. The lens may be any lens, such asthe first camera lens 606, the second camera lens 608, the camera lens704, the first camera lens 806, the second camera lens 808, the firstlens 906, the second lens 908, the first lens 1106, the second lens1108, the first lens 1206, the second lens 1208, the lens 1660, the lens2015, or another lens. Since the right portion of the scene 1002 iscloser to the image sensor 1004 than the left portion of the scene 1002,the captured image frame 1006 includes a perspective distortion. Theperspective distortion is shown as the right portion of the scene 1002in the image frame 1006 appearing larger than the left portion of thescene 1002 in the image frame 1006. Since the angle of the image sensor1004 with reference to another image sensor is known (such as betweenimage sensors 602 and 604 in the conceptual depiction in FIG. 6), thedevice 500 (such as the image signal processor 512) may perform aperspective distortion correction 1022 to generate the processed image1008. The device 500 may modify the captured image frame 1006 using theperspective distortion correction 1022 to generate the processed image1008. For instance, during perspective distortion correction 1022, thedevice 500 may map a trapezoidal area of the captured image frame 1006onto a rectangular area (or vice versa), which may be referred to as akeystone perspective distortion correction, a keystone projectiontransformation, or keystoning. In some cases, perspective distortioncorrection 1022 may be referred to as perspective distortion,perspective transformation, projection distortion, projectiontransformation, transformation, warping, or some combination thereof.

In capturing the scene 1002, the image sensor 1004 may also captureareas outside of the scene 1002 (such as illustrated by the whitetriangles in the image frame 1006 from the sensor). In someimplementations of a perspective distortion correction 1022, the device500 processes the captured image frame 1006 so that the resultingprocessed image 1008 includes just the illustrated portions of the scene1002, without the additional captured scene information in capturedimage frame 1006. The device 500 takes the left portion of the capturedimage frame 1006 including the illustrated portion of the scene 1002(excluding the additional portions of the captured scene above and belowthe scene 1002 as illustrated by the white triangles) and adjusts theremainder of the captured image frame 1006 to the left portion of thescene 1002 in captured image frame 1006 to generate image 1008. Theportion taken from the left of the captured image frame 1006(corresponding to the illustrated portion of the scene 1002) may bebased on a field of view of the image sensor, the common perspective towhich the captured image frame 1006 is to be adjusted, and theperspective of the other image sensor capturing a different portion ofthe scene not illustrated. For example, based on the two perspectives ofthe cameras, the common perspective, and the field of view, the device500 may use a range of image pixels in the left column of image pixelsof the captured image frame 1006 for the processed image 1008.

Similarly, the portion taken from the right of the image frame 1006(corresponding to the illustrated portion of the scene 1002) may bebased on a field of view of the image sensor, the common perspective towhich the image frame 1006 is to be adjusted, and the perspective of theother image sensor capturing a different portion of the scene notillustrated. For example, based on the two perspectives of the cameras,the common perspective, and the field of view, the device 500 may use arange of image pixels in the right column of image pixels of thecaptured image frame 1006 for the processed image 1008. In the examplecaptured image frame 1006, all of the pixels in the furthest rightcolumn of the captured image frame 1006 include information from theillustrated portion of the scene 1002 (the white triangles indicatingadditional portions of the captured scene captured in the captured imageframe 1006 end at the right column of image pixels in image frame 1006).

As shown, the illustrated portion of the scene 1002 is skewed in imageframe 1006 from the smaller range of image pixels in the left column ofimage pixels of the image frame 1006 to the larger range of image pixelsin the right column of image pixels of the image frame 1006. The rate atwhich the number of pixels in the range increase when moving through thecolumns of image pixels from left to right may be linear (which thedevice 500 may determine based on a linear regression of range of pixelsbased on the column or a defined mapping of range of pixels at eachcolumn). In this manner, the image pixels in a column of image pixels ofthe image frame 1006 to be used for the processed image 1008 may be amapping based on the distance of the pixel column from the left columnand from the right column. For example, if the image frame 1006 includes100 columns of 100 pixels of scene information to be used for the image1008 and the left column includes 50 pixels of scene information to beused for the image 1008, the 50th column may include approximately 75pixels of scene information to be used for the image 1008(0.5*50+0.5*100). In addition, the pixels of scene information to beused for the processed image 1008 may be centered at the center of thecolumn of the image frame 1006. Continuing the previous example, the50th column may include 12 or 13 pixels at the bottom of the column notto be used and may include 13 or 12 pixels at the top of the column notto be used.

Based on the desired common perspective for a combined image, the devicemay adjust the pixel values of a captured image frame (such as imageframe 1006) using the selected pixels of scene information to generatethe processed image 1008. The device 500 may generate the combined imagein response to modification of the captured image frame 1006 to generatethe processed image 1008. Adjusting the pixel values causes thehorizontal lines that are parallel in the scene 1002 (which are shown asslanted to one another in the image frame 1006 because of perspectivedistortion) to again be parallel in the image 1008. To adjust pixelvalues for the image 1008 (so that, in the example, the horizontal linesare parallel in the image 1008), the device 500 may “stretch” pixelvalues in the image frame 1006 to cover multiple pixels. For example,stretching a pixel value in the image frame 1006 to cover multiplepixels values in the processed image 1008 may include using the pixelvalue at multiple pixel locations in the image 1008. Conversely, thedevice 500 may combine multiple pixel values in the image frame 1006 tobe used for fewer pixel values in the image 1008 (such as by averagingor other combinatorial means). A binning or a filtering based (such asan averaging, median filtering, and so on) perspective distortioncorrection 1022 process may be applied to pixel values to adjust thecaptured image of the scene 1002 in image frame 1006 to generate theprocessed image 1008. In the example, the process is illustrated asbeing performed in the vertical direction. However, the process may alsobe applied in the horizontal direction to prevent the scene 1002 fromappearing stretched in the processed image 1008. While some examplefilters for perspective distortion correction 1022 are described, anysuitable filter may be used to combine pixel values to generate theprocessed image 1008 in the correction of perspective distortion. As aresult of the perspective distortion correction, the processed image1008 may be horizontally and/or vertically smaller or larger than theimage frame 1006 (in terms of number of pixels).

While the implementations above describe determining a portion of animage frame to be adjusted in correcting perspective distortion, in someimplementations, one or more image sensors may be configured to adjustthe readout for an image frame based on a perspective distortioncorrection. For example, an image sensor 1004 may be configured toreadout from specific image sensor pixels (such as excluding imagesensor pixels capturing scene information in the white triangles ofimage frame 1006). In some implementations, a device may be configuredto adjust which lines (or line portions) of pixels of the image sensorare to be readout based on the portion of the scene 1002 to be includedin the processed image 1008. Perspective distortion may then beperformed on the image frame (which includes only a subset of pixel datafrom the image sensor 1004). The perspective distortion function may bebased on the number of pixels readout from the image sensor. Since imageframes from both cameras include perspective distortion with referenceto the intended perspective for the combined image, the device 500 mayperform perspective distortion correction on image frames from bothcameras.

FIG. 10B is a conceptual diagram 1020 illustrating an exampleperspective distortion correction 1022 of two image frames 1024 to acommon perspective for a combined image 1026. As shown in the two imageframes 1024, the first image and the second image have a perspectivedistortion opposite one another. The device 500 is to correct theperspective distortion (using perspective distortion correction 1022) ofeach of the first image and the second image (such as described above)to a common perspective (such as shown in the combined image 1026).After correcting the perspective distortion, the device 500 may stitchcorrected image 1 and corrected image 2 to generate the combined(wide-angle) image.

Stitching may be any suitable stitching process to generate the combinedimage. In some implementations, the field of view of the first camera501 overlaps the field of view of the second camera 502. For example,the first camera 501, the second camera 502, and the one or moreredirection elements 503 are arranged so that the fields of view overlapby ½ of a degree to 5 degrees. After correcting the perspectivedistortion, the device 500 uses the overlapping portions in the capturedframes from the two cameras 501 and 502 to align and combine the twoimage frames to generate the combined image. Since an overlap exists,the device 500 may reduce stitching errors based on aligning thecaptured image frames. In some implementations, the device 500 maycompensate for a change in overlap over time (such as if the device 500is dropped or bumped, repeated temperature changes cause shifts in oneor more components, and so on). For example, an overlap may begin at 5degrees at device production, but over time, the overlap may increase to7 degrees. The device 500 may use object detection and matching in theoverlapping scene portion of the two image frames to align the imageframes and generate the combined image (instead of using a staticmerging filter based on a fixed overlap and arrangement of components).Through alignment and matching of objects in the overlapping sceneportion of two image frames, the device 500 may use any overlap (as longas of sufficient size, such as ½ of a degree) to stitch the image framestogether to generate the combined image.

FIG. 10C is a conceptual diagram 1040 illustrating an example digitalalignment and stitching 1042 of two image frames captured by two camerasto generate a wide angle image. To illustrate operations of digitalalignment and stitching, the scene is depicted as two instances of theEnglish alphabet (from A-Z twice). The right instance of the alphabet inthe scene is illustrated with each of its letters circled. The leftinstance of the alphabet in the scene with no circle around any of itsletters. Camera 1 (such as the first camera 501) captures the leftinstance of the alphabet in the scene. Camera 2 (such as the secondcamera 502) captured the right instance of the alphabet in the scene.The overlapping fields of view of the two cameras may cause both camerasto capture the “Z{circle around (A)}D” (with the letter “A” circled) inthe middle of the scene. The overlap is based on the angle between thetwo cameras (such as illustrated by virtual lenses and image sensors forlens 906 and sensor 902 for one camera and lens 908 and sensor 904 forthe other camera in FIG. 9). The device 500 performs digital alignmentand stitching 1042 by using object or scene recognition and matchingtowards the right edge of camera 1's image frame and towards the leftedge of camera 2's image frame to align the matching objects/scene.Alignment may include shifting and/or rotating one or both image frameswith reference to the other image frame to overlap pixels between theimage frames until matching objects or portions of the scene overlap.With the image frames aligned based on matching objects/scene, the twoimage frames are stitched together to generate the digital aligned andstitched image (which may include saving the shifted and/or rotatedimage frames together as a combined image). Stitching may includeaveraging overlapping image pixel values, selecting one of the imagepixel values as the combined image pixel value, or otherwise blendingthe image pixel values.

In addition to reducing stitching distortions and reducing perspectivedistortions, the device 500 may reduce a non-uniform brightnessdistortion in a combined image. One or more camera lenses can beconfigured to image the scene onto an image sensor. The relativeillumination of the image formed by the lens can follow a low or minimumof of I(θ)=Io×cos⁴θ, where θ is the angle between the incoming ray andthe normal of the lens, Io is a constant and I(θ) is the illumination ofthe image pixel illuminated by the incoming light at an angle of θ.Light normal to the lens (θ=0) will be focused to the center of thesensor, and light at the largest angle (say θ=30°) will be focused ontothe edge of the sensor). As such, the image brightness at the edge iscos⁴(30°)=0.56 of the brightness at the center. Additionally, the lightredirection components, such as the mirrors in FIG. 8 and the prisms inFIG. 9, may introduce vignetting that may further reduce the brightnessof the image pixels near the edges. As a result, more light may reachthe center of the image sensor than the edges of the image sensor. Notas much light may reach the edges (and especially the corner pixels) ofthe image sensor as the center of the image sensor. Captured imageframes from the first camera 501 and the second camera 502 can thus havea non-uniform brightness across their image pixels. Vignetting or otherbrightness non-uniformities in a first image frame from the first camera501 and/or in a second image frame from the second camera 502 can causea visible seam in a combined image generated by combining the firstimage with the second image. Post-capture (such as before or aftercorrecting the perspective distortion and/or before or after stitchingthe image frames together), the device 500 may correct the brightnessnon-uniformity of the image frames for the combined image. For example,the device 500 may adjust brightness in a first image frame from thefirst camera 501 to remove vignetting from the first image, may adjustbrightness in a second image frame from the second camera 502 to removevignetting from the second image, or both. The device 500 may make thesebrightness adjustments before the device 500 combines the first imageand the second image to generate the combined image. Removal ofvignetting through such brightness adjustments can ensure that there isno visible seam in the combined image (e.g., between the portion of thecombined image that is from the first image and the portion of thecombined image that is from the second image).

Additionally, in some cases, the first camera 501 and the second camera502, may receive unequal amounts of light, may process light and/orimage data differently (e.g., due to differences in camera hardwareand/or software), and/or may be miscalibrated. Unequal levels ofbrightness or another image property between a first image frame fromthe first camera 501 and a second image frame from the second camera 502can cause a visible seam in a combined image generated by combining thefirst image with the second image. In some examples, the device 500 mayincrease or decrease brightness in a first image frame from the firstcamera 501, may increase or decrease brightness in a second image framefrom the second camera 502, or both. The device 500 may make thesebrightness adjustments before the device 500 combines the first imageand the second image to generate the combined image. Such brightnessadjustments can ensure that there is no visible seam in the combinedimage (e.g., between the portion of the combined image that is from thefirst image and the portion of the combined image that is from thesecond image). FIG. 10D is a conceptual diagram 1060 illustrating anexample brightness uniformity correction 1062 of a wide angle imagegenerated from two image frames captured by two cameras. The brightnessuniformity correction 1062 can correct vignetting or other brightnessnon-uniformities as discussed above with respect to FIG. 10C. Graph 1064shows the relative illumination of the image sensors based on theillumination at the center of the image sensors of the first camera 501and the second camera 502. The center of each image sensor isilluminated the most (indicated by the image sensor centers beingpositioned at a 30 degree angle from the center of the combined image.This angle can be measured between the incoming light and the normal ofthe top surfaces of the prisms discussed herein (e.g., side 922 and side924 in FIG. 9, side 1220 in FIGS. 12A-12C). In some examples, the lensescan be tilted with respect to the prisms' top surface normal by 30degrees, for instance as indicated by the angles of the first virtuallens 926 and the second virtual lens 928 in FIG. 9. An incoming light of30 degree can be normal to the lens, and can thus be focused at thecenter of the sensor and have the largest illumination/brightness in theresulting image. If each image sensor includes a 70 degree angle ofview, the fields of view of the two image sensors may overlap by 10degrees. The illumination of the image sensors decreases when movingfrom the centers of the image sensors (e.g., the centers correspondingto −30 degrees and 30 degrees respectively in the graph 1064) towardsthe edges of the image sensors (e.g., the edges indicated by 0 in themiddle of the graph 1064 and the two ends of the graph 1064). Whilegraph 1064 is shown along one axis of the image sensor for illustrationpurposes, the graph 1064 may include additional dimensions or may begraphed in other ways to indicate the change in illumination based on atwo-dimensional image sensor.

In some implementations, an indication of the illumination of differentportions of the image sensor based on the illumination of the imagesensor center (such as a fraction, decimal or ratio indicating thedifference for each portion) may be determined. For example, the graph1064 may be known based on the type of camera or determined duringcalibration of the camera (with the graph 1064 being embodied to cover atwo dimensional area for the image sensor). In some implementations,graph 1064 can be obtained during a calibration by capturing imageframes of a test scene (such as a scene with a uniform background) usinga uniform illumination. The pixel values of the processed image (withoutuniformity correction) may thus indicate the change in illuminationrelative to a location in the image. With such indications or the graph1064 known for the first camera 501 and the second camera 502, thedevice performs a brightness uniformity correction 1062 to generate animage with a uniform correction (as shown in graph 1066).

In some implementations, the device 500 increases the brightness ofimage pixels in the image frame (such as increasing a luminance value ina YUV color space or similarly increasing RGB values in an RGB colorspace). The amount to increase the brightness of an image pixel may beto divide the current brightness value by the fraction of illuminationbetween the associated image sensor pixel and the image sensor center(such as based on graph 1064). In this manner, each image pixel'sbrightness may be increased to be similar to an image pixel's brightnessof the image sensor center (as shown in graph 1066).

The device 500 may thus generate a combined image including correctedperspective distortion, reduced stitching artifacts, and reducedbrightness distortion (non-uniform brightness) using one or moreredirection elements 503 to direct light to the first camera 501 and thesecond camera 502 for image frame capture.

Some implementations of the one or more redirection elements and camerasmay cause a scattering noise in a combined image.

FIG. 11 is a conceptual diagram 1100 illustrating example lightreflections from a first camera lens 1106 that may cause scatteringnoise in a portion of an image frame. A first camera includes a firstimage sensor 1102 and the first camera lens 1106. The first camera maybe an embodiment of the first camera in FIG. 9 (including a first imagesensor 902 and a first camera lens 906). A first redirection element1110 is positioned outside of the first camera to direct light towardsthe first image sensor 1102. As shown, light received at one side of thefirst redirection element 1110 is refracted by a first prism of thefirst redirection element 1110, reflected by a first mirror on the side1112 of the first prism, and directed towards the camera lens 1106. Thefirst camera lens 1106 may reflect a small portion of the light backtowards the first prism through Fresnel reflection. The light receivedtowards a top end of the image sensor 1102 indicates the remainder ofthe light that is allowed to pass through the lens 1106. The lightreflected by the first camera lens 1106 passes back through the firstprism towards the top-right edge of the prism. The top-right edge of thefirst prism may be referred to as the edge of the first prism that isclosest to the second prism of a second redirection element 1120. Thefirst prism and/or the second prism can include a high refractive indexmedium (e.g., having a refractive index above a threshold). While notshown, one or more edges of a prism of a redirection element may bechamfered (to mitigate cracking). The top-right edge of the prism (whichmay be chamfered) may reflect and scatter the light from the camera lens1106 back towards the camera lens 1106, and the camera lens 1106 maydirect the light towards the bottom end of the image sensor 1102. Inthis manner, light intended for one portion of the image sensor 1102 maybe erroneously received by a different portion of the image sensor 1102.Light received in unintended locations of the image sensor 1102 maycause the first camera to capture image frames with distorted brightnessin the form of scattering noise and related image artifacts. While thescattering noise is only shown for the first camera (with the first lens1106 and first image sensor 1102) and the first redirection element1110, the scattering noise may occur for the second camera (with thesecond lens 1108 and the second image sensor 1104) and the secondredirection element 1120 as well. In addition, the scattering noise mayoccur in the portions of the image sensors corresponding to theoverlapping fields of view for the cameras. Therefore, a combined imagemay include the scattering noise near the stitch line or location of oneside of the combined image. This may result in a visible stitch line inthe combined image, which is not desirable as it breaks the continuityin image data in the combined image.

One or more redirection elements 503 are configured to preventredirecting light from a camera lens back towards the camera lens. Forexample, the redirection elements 1110 may be configured to preventreflecting light from the camera lens 1106 back towards the camera lens1106 (and similar for the other redirection element). In someimplementations, a portion of one or more edges of the prism isprevented from scattering light. In preventing the portions fromscattering light, one or more of the chamfered edges of the prism areprevented from scattering light. For example, a light absorbing coatingmay be applied to the top right chamfered edge of the prism in theexample in FIG. 11). In some implementations, one or both of the othertwo corner edges of the prism (that are not in the illustrated lightpaths in FIG. 11 and which may or may not be chamfered) may also becoated with a light absorbing coating to prevent light scattering fromthe surfaces at these locations. In this manner, light received at thetop-right edge of the left prism in FIG. 11 is absorbed and will not bescattered toward the camera lens 1106 and the sensor 1102. In someexamples, the light absorbing coating may be opaque. In some examples,the light absorbing coating may be black, dark grey, or a dark color.

In some other implementations to reduce the scattering noise caused byreflections from the camera lenses and subsequently scattered by a prismedge, the first redirection element and the second redirection elementmay be combined into a single redirection element so that the top-rightcorner of the left prism and the top-left corner of the right prism areeffectively eliminated (do not physically exist).

FIG. 12A is a conceptual diagram 1200 illustrating an exampleredirection element 1210 to redirect light to a first camera and toredirect light to a second camera. The first camera includes a firstimage sensor 1202 and a first camera lens 1206, and the first camera maybe an example implementation of the first camera in FIG. 9. The secondcamera includes a second image sensor 1204 and a second camera lens1208, and the second camera may be an example implementation of thesecond camera in FIG. 9. For example, the angle of view Theta for bothcameras may be 70 degrees.

The redirection element 1210 includes a first prism 1212 to refractlight intended for the first image sensor 1202 and a second prism 1214to refract light intended for the second image sensor 1204. A firstmirror may be on side 1216 of the first prism 1212, and a second mirrormay be on side 1218 of the second prism 1218 (similar to redirectionelements 910 and 912 in FIG. 9). The first prism 1212 and/or the secondprism 1218 can include a high refractive index medium (e.g., having arefractive index above a threshold). The first prism 1212 and the secondprism 1214 are contiguous. The first prism 1212 and the second prism1214 are physically connected and/or joined and/or bridged at the top ofsides 1216 and 1218. For example, the prisms 1212 and 1214 are connectedso as to be overlapping at a top edge of both prisms. For instance, theedge of the first prism 1212 that is closest to the second prism 1214,and the edge of the second prism 1214 that is closest to the first prism1212, overlap and are joined together. In some implementations, theoverlapping section of prisms 1212 and 1214 may have a height of ½ mm to1 mm of the redirection element 1210. The overlapping section of prisms1212 and 1214 may be referred to as a bridge joining the first prism1212 and the second prism 1214.

In this manner, light received near the center of the side 1220 of theredirection element may be reflected towards the first image sensor 1202or the second image sensor 1204 based on which side 1216 or 1218receives the light. Light reflected back by the camera lens 1206 and thecamera lens 1208 towards the redirection element 1210 does not hit theprism corner edge (as illustrated in FIG. 11) since the prism corneredge does not exist in the redirection element 1210.

In some implementations of manufacturing the redirection element 1210,an injection molding of the desired shape (such as including twocontiguous/overlapping triangular or equilateral triangular prisms) isfilled with a plastic having a desired refractive index. After creatinga plastic element shaped as desired, two surfaces of the plastic elementhave a reflective coating applied (such as sides 1216 and 1218). In someimplementations, an anti-reflective coating is applied to the top sideto receive light from the scene (such as side 1220). An anti-reflectivecoating may also be applied to the sides of the prisms oriented towardsthe camera lenses 1206 and 1208. In some implementations, a proximalside and a distal side of the redirection element 1210 also include anon-reflective and/or light-absorbing coating. In some examples, thecoating may be opaque. In some examples, the coating may be black, darkgrey, or a dark color. With the top corners of the prisms 1212 and 1214closest to each other overlapping, the cameras may be positioned toensure the virtual center of the first lens 1206 and the second lens1208 virtually overlap while remaining physically separate as in FIG. 9(e.g., the center of the first entrance pupil of the first lens 1206 andthe center of the second entrance pupil of the second lens 1208 overlapas in FIG. 9). In some implementations, the orientations of the camerasare the same or similar as in FIG. 9 to ensure 0.5-5 degree overlap ofthe scenes at the center stitch area of the combined image of the twoimages captured by image sensors 1202 and 1204. While not shown in FIG.12A (or the other implementations of a prism of a redirection element),one or more of the corner edges may be chamfered to prevent cracking.

While virtual lenses corresponding to the first lens 1206 and the secondlens 1208 are not illustrated in FIG. 12A, it should be understood thatthe positions of such virtual lenses would be similar to the positionsof the first virtual lens 926 and the second virtual lens 928 of FIG. 9.While virtual image sensors corresponding to the first image sensor 1202and the second image sensor 1204 are not illustrated in FIG. 12A, itshould be understood that the positions of such virtual image sensorswould be similar to the positions of the first virtual image sensor 914and the second virtual image sensor 916 of FIG. 9. While virtualextensions of the prior paths of the first light and the second lightbeyond the first prism 1212 and the second prism 1214 toward the virtuallenses and the virtual image sensors are not illustrated in FIG. 12A, itshould be understood that virtual extensions of the prior paths of thefirst light and the second light in FIG. 12A would appear similarly tothe virtual extensions of the prior paths of the first light and thesecond light in FIG. 9.

FIG. 12B is a conceptual diagram 1240 illustrating the redirectionelement in FIG. 12A that illustrates the elimination of light scatteringfrom a prism edge (such as shown in FIG. 11). A strong side illuminationentering the prism 1212 is refracted and reflected by a reflectivesurface on side 1216. The reflected light exits the redirection element1210 at a refraction angle and continues propagation towards lens 1206.The portion of light reflected from the lens surface through Fresnelreflection re-enters the prism 1212 and propagates towards thetop-center (where the two prisms 1212 and 1214 overlap). Since prism1212 (and prism 1214) does not include a corner edge at the top centerof the redirection element 1210, there is no light scatter back towardsthe lens 1206. For example, the light reflected from the lens 1206 maycontinue to propagate and exit the redirection element 1210 on side1220. A camera may be oriented with reference to the redirection element1210 to ensure subsequent specular reflections from other prism surfaces(such as from side 1220) will not be received by its image sensor. Whilereduction of light scatter is illustrated with reference to prism 1212in FIG. 12B, the same reduction of light scatter may occur for thesecond prism 1214 regarding light reflected by the second camera lens1208 associated with the second image sensor 1204. Because the reflectedlight exits the redirection element 1210 on side 1220, the scatteringnoise and visible seam discussed with respect to FIG. 11 are reduced oreliminated using the redirection element 1210 with the overlappingjoined prisms 1212 and 1214 illustrated in FIGS. 12A-12C. Thus, use ofthe redirection element 1210 with the overlapping joined prisms 1212 and1214 increases image quality, both of images captured individually usingthe image sensors 1202 and 1204, and of combined images generated bystitching together images captured by the image sensors 1202 and 1204.Additionally, prisms 1212 and 1214 are overlapping and joined in theredirection element 1210, the redirection element 1210 has theadditional benefit of ensuring that the prisms 1212 and 1214 can bepositioned relative to one another with precision, and do not getmisaligned relative to one another without need for additional hardwarecontrolling the relative positions of the prisms 1212 and 1214 to oneanother.

FIG. 12C is a conceptual diagram 1260 illustrating the redirectionelement in FIG. 12A from a perspective view. The light redirectionelement 1210 is illustrated in between a first camera and a secondcamera. The first camera includes the first lens 1206, which is hiddenfrom view based on the perspective in the conceptual diagram 1260, butis still illustrated using dashed lines. The second camera includes thesecond lens 1208, which is hidden from view based on the perspective inthe conceptual diagram 1260. The light redirection element 1210 includesthe first prism 1212 and the second prism 1214. The first prism 1212 andthe second prism 1214 are contiguous. The edge of the first prism 1212closest to the second prism 1214 is joined to the edge of the secondprism 1214 closest to the first prism 1212. Side 1216 of the first prism1212 includes a reflective coating. Side 1218 of the second prism 1218includes a reflective coating. The light redirection element 1210includes a side 1220 that is hidden from view based on the perspectivein the conceptual diagram 1260, but is still pointed to using a dashedline.

In some cases, the first prism 1212 may be referred to as a first lightredirection element, and the second prism 1214 may be referred to as asecond light redirection element. In some cases, an edge of the firstlight redirection element physically overlaps with, and is joined to, anedge of the second light redirection element. In some cases, an edge ofthe first prism physically overlaps with, and is joined to, an edge ofthe second prism. In some cases, the first side 1216 (having areflective surface) of the first prism 1212 may be referred to as afirst light redirection element, and the second side 1218 (having areflective surface) of the second prism 1214 may be referred to as asecond light redirection element. The redirection element 1210 may bereferred to as a single light redirection element, where the first lightredirection element and the second light redirection element are twodistinct portions of the single light redirection element.

As shown above, one or more redirection elements may be used indirecting light from a scene towards multiple cameras. The multiplecameras capture image frames to be combined to generate a wide angleimage. Such as wide angle image includes less distortion caused by lenscurvature and may have a wider angle of view than other single camerasfor wide-angle imaging.

Before, concurrently with, contemporaneously with, and/or aftercombining a first image frame and a second image frame to generate acombined image, the device 500 may perform other processing filters onthe combined image or the captured image frames. For example, the imageframes may have different color temperatures or light intensities. Otherexample processing may include imaging processing filters performedduring the image processing pipeline, such as denoising, edgeenhancement, and so on. After processing the image, the device 500 maystore the image, output the image to another device, output the image toa display 514, and so on. In some implementations, a sequence of wideangle images may be generated in creating a wide angle video. Forexample, the image sensors concurrently and/or contemporaneously capturea sequence of image frames, and the device 500 processes the associatedimage frames as described for each in the sequence of image frames togenerate a sequence of combined images for a video. Example methods forgenerating a combined image are described below with reference to FIG.13A, FIG. 13B, and FIG. 14. While the methods are described as beingperformed by the device 500 and/or by an imaging system, any suitabledevice may be used in performing the operations in the examples.

FIG. 13A is a flow diagram illustrating an example process 1300 forgenerating a combined image from multiple image frames. In someexamples, the operations in the process 1300 may be performed by animaging system. In some examples, the imaging system is the device 500.In some examples, the imaging system includes at least one of the camera112, the camera 206, the device 500, the imaging architectureillustrated in conceptual diagram 600, the imaging architectureillustrated in conceptual diagram 700, the imaging architectureillustrated in conceptual diagram 800, the imaging architectureillustrated in conceptual diagram 900, the imaging architectureillustrated in conceptual diagram 1100, the imaging architectureillustrated in conceptual diagram 1200, the imaging architectureillustrated in conceptual diagram 1240, the imaging architectureillustrated in conceptual diagram 1260, the imaging architectureillustrated in conceptual diagram 1600, least one of an image captureand processing system 2000, an image capture device 2005A, an imageprocessing device 2005B, an image processor 2050, a host processor 2052,an ISP 2054, a computing system 2100, one or more network servers of acloud service, or a combination thereof.

At operation 1302, the imaging system may receive a first image frame ofa scene captured by a first camera 501. For example, after the firstcamera 501 captures the first image frame (including a first portion ofthe scene), the image signal processor 512 may receive the first imageframe. The first portion of the scene may be one side of the scene. At1304, the device 500 may also receive a second image frame of the scenecaptured by a second camera 502. For example, after the second camera502 captures the second image frame (including a second portion of thescene), the image signal processor 512 may receive the second imageframe. The second portion of the scene may be the other side of thescene.

At operation 1306, the imaging system may generate a combined image fromthe first image frame and the second image frame. The combined imageincludes a field of view wider than the first image frame's field ofview or the second image frame's field of view. For example, the firstimage frame and the second image frame may be stitched together (asdescribed above). In some implementations, an overlap in the sides ofthe scene captured in the image frames is used to stitch the first imageframe and the second image frame.

The combined image may have parallax effects reduced or removed based onvirtually overlapping the centers of the entrance pupils of the firstcamera 501 and the second camera 502 capturing the first image frame andthe second image frame based on one or more redirection elements 503(such as redirection elements in FIG. 8, 9, or 12A-12C). In this manner,lenses or other components do not physically overlap while the entrancepupils' centers virtually overlap. In some implementations, the imageframes are captured concurrently and/or contemporaneously by cameras 501and 502 to reduce distortions caused by local motion or global motion.

While not shown in FIG. 13A, the imaging system may continue processingthe combined image, including performing denoising, edge enhancement, orany other suitable image processing filter in the image processingpipeline. The resulting combined image may be stored in the memory 506or another suitable memory, may be provided to another device, may bedisplayed on display 514, or may otherwise be used in any suitablemanner.

FIG. 13B is a flow diagram illustrating an example method 1350 ofdigital imaging. In some examples, the operations in the process 1300may be performed by an imaging system. In some examples, the imagingsystem is the device 500. In some examples, the imaging system includesat least one of the camera 112, the camera 206, the device 500, theimaging architecture illustrated in conceptual diagram 600, the imagingarchitecture illustrated in conceptual diagram 700, the imagingarchitecture illustrated in conceptual diagram 800, the imagingarchitecture illustrated in conceptual diagram 900, the imagingarchitecture illustrated in conceptual diagram 1100, the imagingarchitecture illustrated in conceptual diagram 1200, the imagingarchitecture illustrated in conceptual diagram 1240, the imagingarchitecture illustrated in conceptual diagram 1260, the imagingarchitecture illustrated in conceptual diagram 1600, least one of animage capture and processing system 2000, an image capture device 2005A,an image processing device 2005B, an image processor 2050, a hostprocessor 2052, an ISP 2054, a computing system 2100, one or morenetwork servers of a cloud service, or a combination thereof.

At operation 1355, the imaging system receives a first image of a scenecaptured by a first image sensor. A first light redirection elementredirects a first light from a first path to a redirected first pathtoward the first image sensor. The first image sensor captures the firstimage based on receipt of the first light at the first image sensor. Insome examples, the imaging system includes the first image sensor and/orthe first light redirection element. In some examples, the first imagesensor is part of a first camera. The first camera can also include afirst lens. In some examples, the imaging system includes the first lensand/or the first camera.

Examples of the first image sensor of operation 1355 include the imagesensor 106, the image sensor of the camera 206, the image sensor of thefirst camera 501, the image sensor of the second camera 502, the firstimage sensor 602, the second image sensor 604, the image sensor 702, thefirst image sensor 802, the second image sensor 804, the first imagesensor 902, the second image sensor 904, the image sensor 1004, thefirst image sensor 1102, the second image sensor 1104, the first imagesensor 1202, the second image sensor 1204, the image sensor 2030,another image sensor described herein, or a combination thereof.Examples of the first lens of operation 1355 include the lens 104, alens of the camera 206, a lens of the first camera 501, a lens of thesecond camera 502, the first camera lens 606, the second camera lens608, the camera lens 704, the first camera lens 806, the second cameralens 808, the first lens 906, the second lens 908, the first lens 1106,the second lens 1108, the first lens 1206, the second lens 1208, thelens 1660, the lens 2015, another lens described herein, or acombination thereof. Examples of the first light redirection element ofoperation 1355 include the light redirection element 706, the firstlight redirection element 810, the second light redirection element 812,the first light redirection element 910, the second light redirectionelement 912, the first prism of the first light redirection element 910,the second prism of the second light redirection element 912, the firstreflective surface on side 918 of the light redirection element 910, thesecond reflective surface on side 920 of the second light redirectionelement 912, the first light redirection element 1110, the second lightredirection element 1112, the first prism of the first light redirectionelement 1110, the second prism of the second light redirection element1112, the first reflective surface on side 1112 of the first lightredirection element 1110, the second reflective surface of the secondlight redirection element 1112, the light redirection element 1210, thefirst prism 1212 of the light redirection element 1210, the second prism1214 of the light redirection element 1210, the first reflective surfaceon side 1216 of the light redirection element 1210, the secondreflective surface on side 1218 of the second light redirection element1212, another prism described herein, another reflective surfacedescribed herein, another light redirection element described herein, ora combination thereof.

At operation 1360, the imaging system receives a second image of thescene captured by a second image sensor. A second light redirectionelement redirects a second light from a second path to a redirectedsecond path toward the second image sensor. The second image sensorcaptures the second image based on receipt of the second light at thesecond image sensor. A virtual extension of the first path beyond thefirst light redirection element intersects with a virtual extension ofthe second path intersect beyond the second light redirection element.In some examples, the imaging system includes the second image sensorand/or the second light redirection element. In some examples, thesecond image sensor is part of a second camera. The second camera canalso include a second lens. In some examples, the imaging systemincludes the second lens and/or the second camera.

Examples of the second image sensor of operation 1360 include the imagesensor 106, the image sensor of the camera 206, the image sensor of thefirst camera 501, the image sensor of the second camera 502, the firstimage sensor 602, the second image sensor 604, the image sensor 702, thefirst image sensor 802, the second image sensor 804, the first imagesensor 902, the second image sensor 904, the image sensor 1004, thefirst image sensor 1102, the second image sensor 1104, the first imagesensor 1202, the second image sensor 1204, the image sensor 2030,another image sensor described herein, or a combination thereof.Examples of the second lens of operation 1360 include the lens 104, alens of the camera 206, a lens of the first camera 501, a lens of thesecond camera 502, the first camera lens 606, the second camera lens608, the camera lens 704, the first camera lens 806, the second cameralens 808, the first lens 906, the second lens 908, the first lens 1106,the second lens 1108, the first lens 1206, the second lens 1208, thelens 1660, the lens 2015, another lens described herein, or acombination thereof. Examples of the second light redirection element ofoperation 1360 include the light redirection element 706, the firstlight redirection element 810, the second light redirection element 812,the first light redirection element 910, the second light redirectionelement 912, the first prism of the first light redirection element 910,the second prism of the second light redirection element 912, the firstreflective surface on side 918 of the light redirection element 910, thesecond reflective surface on side 920 of the second light redirectionelement 912, the first light redirection element 1110, the second lightredirection element 1112, the first prism of the first light redirectionelement 1110, the second prism of the second light redirection element1112, the first reflective surface on side 1112 of the first lightredirection element 1110, the second reflective surface of the secondlight redirection element 1112, the light redirection element 1210, thefirst prism 1212 of the light redirection element 1210, the second prism1214 of the light redirection element 1210, the first reflective surfaceon side 1216 of the light redirection element 1210, the secondreflective surface on side 1218 of the second light redirection element1212, another prism described herein, another reflective surfacedescribed herein, another light redirection element described herein, ora combination thereof.

In some examples, the first lens and the second lens virtually overlap.In some examples, while the first lens and the second lens virtuallyoverlap, the first lens and second lens do not physically overlap, donot spatially overlap, are physically separate, and/or are spatiallyseparate. For example, the first lens 906 and the second lens 908 ofFIG. 9 do not physically overlap, do not spatially overlap, arephysically separate, and are spatially separate. Despite this, the firstlens 906 and the second lens 908 virtually overlap, since the firstvirtual lens 926 (the virtual position of the first lens 906) overlapswith the second virtual lens 928 (the virtual position of the secondlens 908). Though virtual lens positions for the first lens 1106 and thesecond lens 1108 are not illustrated in FIG. 11, the first lens 1106 andthe second lens 1108 can also virtually overlap (e.g., the virtual lensposition of the first lens 1106 can overlap with the virtual lensposition of the second lens 1108). The first lens 1106 and the secondlens 1108 do not physically overlap, do not spatially overlap, arephysically separate, and are spatially separate. Though virtual lenspositions for the first lens 1206 and the second lens 1208 are notillustrated in FIGS. 12A-12C, the first lens 1206 and the second lens1208 can also virtually overlap (e.g., the virtual lens position of thefirst lens 1206 can overlap with the virtual lens position of the secondlens 1208). The first lens 1206 and the second lens 1208 do notphysically overlap, do not spatially overlap, are physically separate,and are spatially separate.

The first light redirection element can include a first reflectivesurface. Examples of the first reflective surface can include thereflective surface of the redirection element 706, the reflectivesurface of the first light redirection element 810, the reflectivesurface on side 918 of the first light redirection element 910, thereflective surface on side 1112 of the first light redirection element1110, the reflective surface on side 1216 of the light redirectionelement 1210, another reflective surface described herein, or acombination thereof. To redirect the first light toward the first imagesensor, the first light redirection element uses the first reflectivesurface to reflect the first light toward the first image sensor.Similarly, the second light redirection element can include a secondreflective surface. Examples of the second reflective surface caninclude the reflective surface of the redirection element 706, thereflective surface of the second light redirection element 812, thereflective surface on side 920 of the second light redirection element912, the reflective surface on side of the second light redirectionelement 1120 closest to 1112 of the first light redirection element1110, the reflective surface on side 1218 of the light redirectionelement 1210, another reflective surface described herein, or acombination thereof. To redirect the second light toward the secondimage sensor (e.g., second image sensor 904/1204), second lightredirection element uses the second reflective surface to reflect thesecond light toward the second image sensor. The first reflectivesurface can be, or can include, a mirror. The second reflective surfacecan be, or can include, a mirror.

The first light redirection element can includes a first prismconfigured to refract the first light. The second light redirectionelement can include a second prism configured to refract the secondlight. In some examples, the first prism and the second prism arecontiguous (e.g., as in FIGS. 12A-12C). For instance, the first prismand the second prism may be made of a single piece of plastic, glass,crystal, or other material. A bridge may join a first edge of the firstprism and a second edge of the second prism. For instance, in FIGS.12A-12C, the edge of the first prism between side 1220 and the side 1216is joined, via a bridge, to the edge of the second prism between side1220 and side 1218. The bridge can be configured to prevent reflectionof light from at least one of first edge of the first prism and thesecond edge of the second prism. For instance, as illustrated in FIGS.12A-12C, the bridge joining the two prisms may prevent the scatteringfrom the prism corner that is illustrated and labeled in FIG. 11.

The first prism can include at least one chamfered edge. For instance,in the first redirection element 910 of FIG. 9, the edge between side922 and side 918 can be chamfered. The corresponding edge of the firstprism in the first redirection element 1110 of FIG. 11 can be chamfered.The second prism can include at least one chamfered edge. For instance,in the second redirection element 912 of FIG. 9, the edge between side924 and side 920 can be chamfered. The corresponding edge of the secondprism in the second redirection element 1120 of FIG. 11 can bechamfered. The first prism can include at least one edge with alight-absorbing coating. For instance, in the first redirection element910 of FIG. 9, the edge between side 922 and side 918 can have alight-absorbing coating. The corresponding edge of the first prism inthe first redirection element 1110 of FIG. 11 can have a light-absorbingcoating. The corresponding edge of the first prism 1212 in theredirection element 1210 of FIGS. 12A-12C (e.g., at and/or near thebridge joining the first prism 1212 with the second prism 1214) can havea light-absorbing coating. The second prism can include at least oneedge with the light-absorbing coating. For instance, in the secondredirection element 912 of FIG. 9, the edge between side 924 and side920 can have a light-absorbing coating. The corresponding edge of thesecond prism in the second redirection element 1120 of FIG. 11 can havea light-absorbing coating. The corresponding edge of the second prism1214 in the redirection element 1210 of FIGS. 12A-12C (e.g., at and/ornear the bridge joining the first prism 1212 with the second prism 1214)can have a light-absorbing coating. The light-absorbing coating can be apaint, a lacquer, a material, or another type of coating. Thelight-absorbing coating can be opaque. The light-absorbing coating canbe reflective or non-reflective. The light-absorbing coating can beblack, dark grey, a dark color, a dark gradient, a dark pattern, or acombination thereof.

In some examples, the first path referenced in operations 1355 and 1360refers to a path of the first light before the first light enters thefirst prism. Thus, the first path can be a path that has not yet beenrefracted by the first prism. For instance, in the context of FIG. 9,the first path may refer to the path of the first light before reachingthe top side 922 of the first redirection element 910. In the context ofFIG. 11, the first path may refer to the path of the first light beforereaching the corresponding top side (not labeled) of the firstredirection element 1110. In the context of FIGS. 12A-12C, the firstpath may refer to the path of the first light before reaching thecorresponding top side 1220 of the first prism 1212 of the redirectionelement 1210. In some examples, the second path referenced in operations1355 and 1360 refers to a path of the second light before the secondlight enters the second prism. Thus, the second path can be a path thathas not yet been refracted by the second prism. For instance, in thecontext of FIG. 9, the second path may refer to the path of the secondlight before reaching the top side 924 of the second redirection element912. In the context of FIG. 11, the second path may refer to the path ofthe second light before reaching the corresponding top side (notlabeled) of the second redirection element 1120. In the context of FIGS.12A-12C, the second path may refer to the path of the second lightbefore reaching the corresponding top side 1220 of the second prism 1214of the redirection element 1210.

In some examples, the first prism includes a first reflective surfaceconfigured to reflect the first light. In some examples, the secondprism includes a second reflective surface configured to reflect thesecond light. The first reflective surface can be, or can include, amirror. The second reflective surface can be, or can include, a mirror.In some examples, the first path referenced in operations 1355 and 1360refers to a path of the first light after the first light enters thefirst prism but before the first reflective surface reflects the firstlight. Thus, the first path can already be refracted by the first prism,but not yet reflected by the first reflective surface. For instance, inthe context of FIG. 9, the first path may refer to the path of the firstlight after passing through the top side 922 of the first redirectionelement 910 and entering the first redirection element 910 but beforereaching the reflective surface on side 918 of the first redirectionelement 910. In the context of FIG. 11, the first path may refer to thepath of the first light after entering the first redirection element1110 but before reaching the reflective surface on side 1112 of thefirst redirection element 1110. In the context of FIGS. 12A-12C, thefirst path may refer to the path of the first light after passingthrough the top side 1220 of the first prism 1212 of the redirectionelement 1210 and entering the first prism 1212 of the redirectionelement 1210 but before reaching the reflective surface on side 1216 ofthe first prism 1212 of the redirection element 1210. In some examples,the second path referenced in operations 1355 and 1360 refers to a pathof the second light after the second light enters the second prism butbefore the second reflective surface reflects the second light. Thus,the second path can already be refracted by the second prism, but notyet reflected by the second reflective surface. For instance, in thecontext of FIG. 9, the second path may refer to the path of the secondlight after passing through the top side 924 of the second redirectionelement 912 and entering the second redirection element 912 but beforereaching the reflective surface on side 920 of the second redirectionelement 912. In the context of FIG. 11, the second path may refer to thepath of the second light after entering the second redirection element1120 but before reaching the reflective surface on the side of thesecond redirection element 1120 that is closest to the side 1112 of thefirst redirection element 1110. In the context of FIGS. 12A-12C, thesecond path may refer to the path of the second light after passingthrough the top side 1220 of the second prism 1214 of the redirectionelement 1210 and entering the second prism 1214 of the redirectionelement 1210 but before reaching the reflective surface on side 1218 ofthe second prism 1214 of the redirection element 1210.

In some examples, the first image and the second image are capturedcontemporaneously, concurrently, simultaneously, within a shared timewindow, within a threshold duration of time of one another, or acombination thereof. The first light redirection element can be fixedand/or stationary relative to the first image sensor. The second lightredirection element can be fixed and/or stationary relative to thesecond image sensor. The first light redirection element can be fixedand/or stationary relative to the second light redirection element. Thefirst light redirection element can be is fixed and/or stationaryrelative to a housing of the imaging system. The second lightredirection element can be is fixed and/or stationary relative to thehousing of the imaging system. For instance, the first image sensor, thefirst light redirection element, the second image sensor, and the secondlight redirection element can be arranged in a fixed and/or stationaryarrangement as in the various image sensors and light redirectionelements depicted in FIG. 8, FIG. 9, FIG. 11, FIGS. 12A-12C, variants ofthese described herein, or a combination thereof. The first lightredirection element can in some cases be movable relative to the firstimage sensor and/or the second light redirection element and/or ahousing the imaging system, for instance using a motor and/or anactuator. The second light redirection element can in some cases bemovable relative to the second image sensor and/or the first lightredirection element and/or a housing the imaging system, for instanceusing a motor and/or an actuator.

A first planar surface of the first image sensor can face a firstdirection, and a second planar surface of the second image sensor canface a second direction. The first direction may be an optical axis ofthe first image sensor and/or of a lens associated with the first imagesensor and/or of a camera associated with the first image sensor. Thesecond direction may be an optical axis of the second image sensorand/or of a lens associated with the second image sensor and/or of acamera associated with the second image sensor. The first direction andthe second direction can be parallel to one another. The first cameracan face the first direction as well. The second camera can face thesecond direction as well. The first direction and the second directioncan point directly at one another. In some examples, the first planarsurface of the first image sensor can face the second planar surface ofthe second image sensor. In some examples, the first camera can face thesecond camera. For example, the first image sensor 802 and the secondimage sensor 804 of FIG. 8 face one another, and face directions thatare parallel to each other's respective directions. The first imagesensor 902 and the second image sensor 904 of FIG. 9 face one another,and face directions that are parallel to each other's respectivedirections. The first image sensor 1102 and the second image sensor 1104of FIG. 11 face one another, and face directions that are parallel toeach other's respective directions. The first image sensor 1202 and thesecond image sensor 1204 of FIGS. 12A-12C face one another, and facedirections that are parallel to each other's respective directions.

At operation 1365, the imaging system modifies at least one of the firstimage and the second image using a perspective distortion correction.The perspective distortion correction of operation 1365 may be referredto as perspective distortion. Examples of the perspective distortioncorrection of operation 1365 include the perspective distortioncorrection 1022 of FIG. 10A, the perspective distortion correction 1022of FIG. 10B, the flat perspective distortion correction 1515 of FIG. 15,the curved perspective distortion correction 1525 of FIG. 15, the flatperspective distortion correction 1620 of FIG. 16, the curvedperspective distortion correction 1630 of FIG. 16, another type ofperspective distortion correction described herein, another type ofperspective distortion described herein, or a combination thereof.

In some examples, to perform the modification(s) of operation 1365 of atleast one of the first image and the second image, the imaging systemmodifies the first image from depicting a first perspective to depictinga common perspective using the perspective distortion correction. Theimaging system modifies the second image from depicting a secondperspective to depicting the common perspective using the perspectivedistortion correction. The common perspective can be between the firstperspective and the second perspective. For instance, in FIG. 10B, thefirst image of the two images 1024 has its perspective angled to theright, while the second image of the two images 1024 has its perspectiveangled to the left. The common perspective, as visible in the firstimage portion of the combined image 1026 and the second image portion ofthe combined image 1026 is straight ahead, in between the right and leftangles of the two images 1024. In FIG. 16, the first original imageplane 1614 has its perspective angled slightly counter-clockwise, whilethe second original image plane 1616 has its perspective angled slightlyclockwise. The common perspective, as visible in the flatperspective-corrected image plane 1625 (as mapped using the flatprojective transformation pixel mapping 1620) is perfectly horizontal,in between the slightly counter-clockwise and slightly clockwise anglesof the first original image plane 1614 and the second original imageplane 1616.

In some examples, to perform the modification(s) of operation 1365 of atleast one of the first image and the second image, the imaging systemidentifies depictions of one or more objects in image data (of the firstimage and/or the second image). The imaging system modifies the imagedata by projecting the image data based on the depictions of the one ormore objects. In some examples, the imaging system can project the imagedata onto a flat perspective-corrected image plane (e.g., as part of aflat perspective distortion correction 1022/1520/1620 as in FIGS.10A-10B, 15, and 16). In some examples, the imaging system can projectthe image data onto a curved perspective-corrected image plane (e.g., aspart of a curved perspective distortion correction 1530/1630 as in FIGS.15, 16, 17, 18, and 19). For instance, in reference to FIG. 15, theimaging system (e.g., the dual-camera device 1505) identifies depictionsthe soda cans in the first image and second image. In the curvedperspective distortion correction 1525, the imaging system (e.g., thedual-camera device 1505) modifies the image data by projecting the imagedata based on the depictions of the soda cans. In reference to FIG. 16,the imaging system (e.g., including the lens 1660) identifies depictionsof one or more objects following a curve in the scene 1655 in the firstimage and second image. In the curved perspective distortion correction1630, the imaging system (e.g., including the lens 1660) modifies theimage data by projecting the image data based on the depictions of theone or more objects following a curve in the scene 1655. In reference toFIG. 17, the imaging system (not pictured) identifies depictions of oneor more objects (e.g., TV 1740, couch 1750) in the scene 1655 in thefirst image and second image. In the different perspective distortioncorrections of the three combined images 1710-1730, the imaging systemcan modify the image data by projecting the image data based on thedepictions of the one or more objects (e.g., TV 1740, couch 1750).

In some examples, the imaging system modifies at least one of the firstimage and the second image using a brightness uniformity correction. Forinstance, the imaging system can remove vignetting and/or otherbrightness non-uniformities from the first image, the second image, orboth. The brightness uniformity correction 1062 of FIG. 10D is anexample of the brightness uniformity correction that the imaging systemcan use to modify the first image and/or the second image. The imagingsystem can also increase or decrease overall brightness in the firstimage, the second image, or both, so that overall brightness matchesbetween the first image and second image. The imaging system can alsoincrease or decrease other image properties (e.g., contrast, colorsaturation, white balance, black balance, color levels, histogram, etc.)in the first image, the second image, or both, so that these imageproperties match between the first image and second image. Suchadjustments of brightness and/or other image properties can ensure thatthere is no visible seam in the combined image (e.g., between theportion of the combined image that is from the first image and theportion of the combined image that is from the second image). In someexamples, the imaging system can perform the modifications relating tobrightness uniformity correction after the modifications relating toperspective distortion correction of operation 1365. In some examples,the imaging system can perform the modifications relating to brightnessuniformity correction before the modifications relating to perspectivedistortion correction of operation 1365. In some examples, the imagingsystem can perform the modifications relating to brightness uniformitycorrection contemporaneously with the modifications relating toperspective distortion correction of operation 1365.

At operation 1370, the imaging system generates a combined image fromthe first image and the second image. The imaging system can generatethe combined image from the first image and the second image in responseto the modification of the at least one of the first image and thesecond image using the perspective distortion correction. The imagingsystem can generate the combined image from the first image and thesecond image in response to the modification of the at least one of thefirst image and the second image using the brightness uniformitycorrection. The combined image includes a combined image field of viewthat is larger than at least one of a first field of view of the firstimage and a second field of view of the second image. For example, thecombined image 1026 of FIG. 10B has a larger and/or wider field of viewthan a first field of view and a second field of view of the first andsecond images in the two images 1024. Similarly, the combined image ofFIG. 10C has a larger and/or wider field of view than a first field ofview and a second field of view of the first image captured by the firstcamera and second image captured by the second camera.

Generating the combined image from the first image and the second imagecan include aligning a first portion of the first image with a secondportion of the second image. Generating the combined image from thefirst image and the second image can include stitching the first imageand the second image together based on the first portion of the firstimage and the second portion of the second image being aligned. Thedigital alignment and stitching 1042 of FIG. 10C are an example of thisalignment and stitching. The first portion of the first image and thesecond portion of the second image can at least partially match. Forexample, in reference to FIG. 10C, the first portion of the first imagemay be the portion of the first image captured by the first camera thatincludes the “Z{circle around (A)}D” (with the letter “A” circled) inthe middle of the scene of FIG. 10C, and the second portion of thesecond image may be the portion of the second image captured by thesecond camera that includes the “Z{circle around (A)}D” (with the letter“A” circled) in the middle of the scene of FIG. 10C. The first portionof the first image and the second portion of the second image can matchcan overlap for stitching. The combined image can include the firstportion of the first image, the second portion of the second image, or amerged image portion that merges or combines image data from the firstportion of the first image with image data from the second portion ofthe second image.

As noted above, the imaging system may be the device 500. The device 500may include at least the first camera 501 and the second camera 502configured to capture the image frames for generating the combinedimage. The device 500 may also include the one or more redirectionelements 503.

FIG. 14 is a flow diagram illustrating an example process 1400 forcapturing multiple image frames to be combined to generate a combinedimage frame. The operations in FIG. 14 may be an example implementationof the operations in FIG. 13A and/or FIG. 13B to be performed by thedevice 500. For example, the device 500 may use a configuration ofcameras and redirection elements depicted in FIG. 8, 9, or 12A-12C (orother suitable redirection elements) to virtually overlap centers ofentrance pupils of the first camera 501 and the second camera 502 (suchas depicted in FIG. 6). Dashed boxes illustrate optional steps that maybe performed.

At operation 1402, a first light redirection element redirects a firstlight towards the first camera 501. For example, a first lightredirection element may redirect a portion of light received from anopening in the device. In some implementations, a first mirror of thefirst light redirection element reflects the first light towards thefirst camera 501 (operation 1404). In the example of FIG. 8, a mirror ofthe first light redirection element 810 may reflect the light from afirst portion of the scene to the first camera lens 806. In the exampleof FIG. 9, the mirror on side 918 of the first prism may reflect thelight from the first portion of the scene to the first camera lens 906.In the example of FIG. 12A, the mirror on side 1216 of the first prism1212 of the redirection element 1210 may reflect the light from thefirst portion of the scene to the first camera lens 1206.

In some implementations, a first prism of the first light redirectionelement may also refract the first light (operation 1406). Referringback to the example of FIG. 9, a redirection element may include both amirror and a prism. For example, a side of a triangular prism mayinclude a reflective coating to reflect light passing through the prism.Referring back to the example of FIG. 12A, a redirection element mayinclude multiple prisms, with one prism to refract the first light forthe first camera 501.

In some implementations, a first lens directs the first light from thefirst light redirection element towards the first camera 501 (operation1408). At operation 1410, the first camera 501 captures a first imageframe based on the first light. At operation 1412, a second lightredirection element redirects a second light towards the second camera502. For example, a second light redirection element may redirect aportion of light received from the opening in the device. In someimplementations, a second mirror of the second light redirection elementreflects the second light towards the second camera 502 (operation1414). In the example of FIG. 8, a mirror of the second redirectionelement 812 may reflect the light from a second portion of the scenetowards the second camera lens 808. In the example of FIG. 9, the secondmirror on side 920 of the second prism of the second redirection element912 may reflect the light from the second portion of the scene to thesecond lens 908. In the example of FIG. 12A, the second mirror on side1218 of the second prism of the redirection element 1210 may reflect thelight from the second portion of the scene to the second lens 1208. Insome implementations, a second prism of the second light redirectionelement may also refract the second light (operation 1416). Referringback to the example of FIG. 9, the second redirection element 912 mayinclude both a mirror and a prism. For example, a side of a triangularprism may include a reflective coating to reflect light passing throughthe prism. Referring back to the example of FIG. 12A, the redirectionelement 1210 may include a second prism and second mirror for reflectingand refracting light towards the second camera lens 1208. Referring backto FIG. 14, in some implementations, the first redirection element andthe second redirection element are the same redirection element. In someimplementations, the redirection element includes multiple prisms andmirrors to redirect the first light and to redirect the second light.For example, the redirection element 1210 in FIG. 12A includes twotriangular prisms 1212 and 1214 (such as equilateral triangular prisms)with mirrors on sides 1216 and 1218.

In some implementations, a second lens may direct the second light fromthe second light redirection element towards an image sensor of thesecond camera 502 (operation 1418). At operation 1420, the second camera502 captures a second image frame based on the second light. As notedabove, the first light redirection element and the second lightredirection element (which may be separate or a single redirectionelement) may be positioned to allow the centers of the entrance pupilsof the first camera 501 and the second camera 502 to virtually overlap.In this manner, parallax effects in the combined image may be reduced orremoved. In some implementations, the second image frame is capturedconcurrently and/or contemporaneously with the first image frame. Inthis manner, multiple image frames may be concurrently and/orcontemporaneously captured by the first camera 501 and the second camera502 of the device 500 to reduce distortions in a combined image causedby global motion or local motion. The captured image frames may beprovided to other components of the device 500 (such as the imageprocessor 512) to process the image frames, including combining theimage frames to generate a combined (wide angle) image in operation1422, as described above).

An image frame as discussed herein can be referred to as an image, animage frame, a video frame, or a frame. An image as discussed herein canbe referred to as an image, an image frame, a video frame, or a frame. Avideo frame as discussed herein can be referred to as an image, an imageframe, a video frame, or a frame. A frame as discussed herein can bereferred to as an image, an image frame, a video frame, or a frame.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof, unless specificallydescribed as being implemented in a specific manner. Any featuresdescribed as modules or components may also be implemented together inan integrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a non-transitory processor-readable storagemedium (such as the memory 506 in the example device 500 of FIG. 5)comprising instructions 508 that, when executed by the processor 504 (orthe camera controller 510 or the image signal processor 512 or anothersuitable component), cause the device 500 to perform one or more of themethods described above. The non-transitory processor-readable datastorage medium may form part of a computer program product, which mayinclude packaging materials.

The non-transitory processor-readable storage medium may comprise randomaccess memory (RAM) such as synchronous dynamic random access memory(SDRAM), read only memory (ROM), non-volatile random access memory(NVRAM), electrically erasable programmable read-only memory (EEPROM),FLASH memory, other known storage media, and the like. The techniquesadditionally, or alternatively, may be realized at least in part by aprocessor-readable communication medium that carries or communicatescode in the form of instructions or data structures and that can beaccessed, read, and/or executed by a computer or other processor.

The various illustrative logical blocks, modules, circuits, andinstructions described in connection with the embodiments disclosedherein may be executed by one or more processors, such as the processor504 or the image signal processor 512 in the example device 500 of FIG.5. Such processor(s) may include but are not limited to one or moredigital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), application specificinstruction set processors (ASIPs), field programmable gate arrays(FPGAs), or other equivalent integrated or discrete logic circuitry. Theterm “processor,” as used herein may refer to any of the foregoingstructures or any other structure suitable for implementation of thetechniques described herein. In addition, in some aspects, thefunctionality described herein may be provided within dedicated softwaremodules or hardware modules configured as described herein. Also, thetechniques could be fully implemented in one or more circuits or logicelements. A general purpose processor may be a microprocessor, but inthe alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

FIG. 15 is a conceptual diagram 1500 illustrating examples of a flatperspective distortion correction 1515 and a curved perspectivedistortion correction 1525. As discussed previously, perspectivedistortion correction can be used to appear to change the perspective,or angle of view, of the photographed scene. In the case of theperspective distortion correction 1022 of FIG. 10B, the perspectivedistortion correction is 1022 used so that the first image and thesecond image appear to share a common perspective, or a common angle ofview, of the photographed scene.

The perspective distortion correction 1022 illustrated in the conceptualdiagram 1020 of FIG. 10B is an example of a keystone perspectivedistortion correction, which is an example of a flat perspectivedistortion correction 1515. A keystone perspective distortion correctionmaps a trapezoidal area into a rectangular area, or vice versa. A flatperspective distortion correction maps a first flat (e.g., non-curved)two-dimensional area onto a second flat (e.g., non-curved) twodimensional area. The first flat (e.g., non-curved) two-dimensional areaand the second flat (e.g., non-curved) two dimensional area may havedifferent rotational orientations (e.g., pitch, yaw, and/or roll)relative to one another. A flat perspective distortion correction may beperformed using matrix multiplication, in some examples.

A device 500 with one of the dual-camera architectures discussed herein(e.g., as illustrated in diagrams 900, 1100, 1200, 1240, and/or 1260)can produce a high quality combined image of many types of scenes usingflat perspective distortion correction 1515. However, the device 500 canproduce a combined image of certain types of scenes that appearsvisually warped and/or visually distorted when using flat perspectivedistortion correction 1515. For such types of scenes, use of a curvedperspective distortion correction 1525 can produce a combined image withreduced or removed visual warping compared to use of flat perspectivedistortion correction 1515.

For example, the conceptual diagram 1500 illustrates a scene 1510 inwhich five soda cans are arranged in an arc partially surrounding adual-camera device 1505, with each of the five soda cans approximatelyequidistant from the dual-camera device 1505. The dual-camera device1505 is a device 500 with one of the dual-camera architectures discussedherein (e.g., as illustrated in diagrams 900, 1100, 1200, 1240, and/or1260), that generates a combined image of the scene 1510 from two imagesof the scene 1510 respectively captured by the two cameras of thedual-camera device 1505 as discussed herein (e.g., as in the flowdiagrams 1300, 1350, or 1400).

The dual-camera device 1505 uses flat perspective distortion correction1515 to perform perspective correction while generating a first combinedimage 1520. The first combined image 1520 appears visually warped. Forinstance, despite the fact that the five soda cans in the scene 1510 areapproximately equidistant from the dual-camera device 1505, the leftmostand rightmost soda cans in the first combined image 1520 appear largerthan the three central soda cans in the first combined image 1520. Theleftmost and rightmost soda cans in the first combined image 1520 alsoappear warped themselves, with their leftmost and rightmost sidesappearing to have different heights. The leftmost and rightmost sodacans in the first combined image 1520 also appear to be farther apartfrom the three central soda cans in the first combined image 1520 thaneach of the three central soda cans in the first combined image 1520 arefrom one another.

The dual-camera device 1505 uses a curved transformation perspectivedistortion correction 1525 to perform perspective correction whilegenerating a second combined image 1530. The second combined image 1530reduces or removes all or most of the apparent visual warping in thefirst combined image 1520. For instance, the five soda cans in the scene1510 appear more similar in size to one another in the second combinedimage 1530 than in the first combined image 1520. The leftmost andrightmost soda cans also appear less warped themselves in the secondcombined image 1530 than in the first combined image 1520. The spacingbetween all five soda cans in the scene 1510 appears to be moreconsistent in the second combined image 1530 than in the first combinedimage 1520.

The curved perspective distortion correction 1525 may be more optimal touse than the flat perspective distortion correction 1515 in a variety oftypes of scenes. For example, the curved perspective distortioncorrection 1525 may be more optimal to use than the flat perspectivedistortion correction 1515 in panorama scenes of a distant horizoncaptured from a high altitude (e.g., a tall building or mountain).

FIG. 16 is a conceptual diagram illustrating pixel mapping from an imagesensor image plane to a perspective-corrected image plane in a flatperspective distortion correction 1515 and in a curved perspectivedistortion correction 1525. In particular, FIG. 16 includes a firstdiagram 1600 that is based on a dual-camera architecture such as thatillustrated in conceptual diagrams 900, 1100, 1200, 1240, and/or 1260.The first diagram 1600 illustrates virtual beams of light passingthrough the first virtual lens 926 and reaching the first virtual imagesensor 914. The first virtual image sensor 914 is also labeled as thefirst original image plane 1614, as the first original image plane 1614represents the first image captured by the first image sensor902/1102/1202 (not pictured). The first diagram 1600 also illustratesvirtual beams of light passing through the second virtual lens 928 andreaching the second virtual image sensor 916. The second virtual imagesensor 916 is also labeled as the second original image plane 1616, asthe second original image plane 1616 represents the second imagecaptured by the second image sensor 904/1104/1204 (not pictured).

The first diagram 1600 illustrates projective transformation pixelmapping 1620 dashed arrows that perform a flat perspective distortioncorrection 1515. The projective transformation pixel mapping 1620 dashedarrows project through various pixels of the first original image plane1614 onto corresponding pixels of a perspective-corrected image plane1625, and project through various pixels of the second original imageplane 1616 onto corresponding pixels of the perspective-corrected imageplane 1625. The perspective-corrected image plane 1625 represents thecombined image generated by merging the first image with the secondimage after performing the flat perspective distortion correction 1515.

A second diagram 1650 in FIG. 16 illustrates an example of a curvedperspective distortion correction 1525. A scene 1655, which may includeboth flat and curved portions, is photographed using a camera with alens 1660. The lens 1660 may be a physical lens (such as lenses 704,806, 808, 906, 908, 1106, 1108, 1206, and/or 1208), or may be a virtuallens (e.g., such as virtual lenses 710, 926, and/or 928). The cameracaptures an image of the scene 1655, the image captured on the flatimage plane 1665. In some examples, the flat image plane 1665 is anoriginal image plane (e.g., as in the first original image plane 1614and/or the second original image plane 1616) representing capture of theimage at a physical image sensor (such as image sensors 702, 802, 804,902, 904, 1004, 1102, 1104, 1202, and/or 1204) and/or a virtual imagesensor (e.g., such as virtual image sensors 708, 914, and/or 916). Insome examples, the flat image plane 1665 is a flat perspective-correctedimage plane 1625 as in the first diagram 1600. Points along the flatimage plane 1665 are represented by a flat x axis. Points along the flatx axis can be found using the equation x=f·tan(α) for a given angle α.In the second diagram 1650, f is the focal length of the camera. In thesecond diagram 1650, α is the angle of view of the camera, or an anglewithin the angle of view of the camera. The angle of view of the cameramay, for example, be 60 degrees. To perform curved perspectivedistortion correction 1525, pixels from the flat image plane 1665 areprojected onto the curved perspective-corrected image plane 1630. Pointsalong the curved perspective-corrected image plane 1630 are representedby a curved x′ axis. Points along the curved x′ axis can be found usingthe equation x′=f·α. Thus, any point along the curved x′ axis is thesame distance f away from the lens 1660, regardless of angle α.

In performing perspective correction on certain images, more nuancedcontrol over the curvature of the curved perspective-corrected imageplane 1630 may be useful. A more nuanced curved perspective distortioncorrection 1525 may be performed using the equation

$x^{''} = {\frac{f \cdot {\tan( {P \cdot \alpha} )}}{P}.}$

Here, x″ represents a variable-curvature perspective-corrected imageplane that depends on a variable P. In this equation, P is a variablethat can be adjusted to adjust the strength of the curvature of thevariable-curvature perspective-corrected image plane. For example, whenP=1, then x″=f·tan(α), making the curved perspective-corrected imageplane 1630 flat and equivalent to the flat image plane 1665 (and to theflat x axis). When P=0, then x″ is undefined—but the limit of as Papproaches 0 is f·α. Thus, for the purposes of the curved perspectivedistortion correction 1525, x″=f·α when P=0, making thevariable-curvature perspective-corrected image plane strongly curved andequivalent to the curved perspective-corrected image plane 1630 (and tothe curved x′ axis). If P is between 0 and 1, the variable-curvatureperspective-corrected image plane is less curved than the curvedperspective-corrected image plane 1630, but more curved than the flatimage plane 1665. Examples of combined images generated using curvedperspective distortion correction 1525 with a variable-curvatureperspective-corrected image plane and P set to different values areprovided in FIG. 17.

FIG. 17 is a conceptual diagram 1700 illustrating three example combinedimages (1710, 1720, and 1730) of a scene that each have differentdegrees of curvature of curved perspective distortion correction 1525applied. The different degrees of curvature of curved perspectivedistortion correction 1525 are applied by mapping to avariable-curvature perspective-corrected image plane using the equation

$x^{''} = \frac{f \cdot {\tan( {P \cdot \alpha} )}}{P}$

as discussed above.

In particular, the first combined image 1710 is generated by applyingcurved perspective distortion correction 1525 to map image pixels onto astrongly curved perspective-corrected image plane, because P=0. Thesecond combined image 1720 is generated by applying curved perspectivedistortion correction 1525 to map image pixels onto a moderately curvedperspective-corrected image plane, because P=0.8. The third combinedimage 1730 is generated by applying perspective distortion correction1515 to map image pixels onto a flat perspective-corrected image plane,because P=1.

All three combined images (1710, 1720, and 1730) depict the same scene,which among other things, depicts a person sitting in a chair facing aTV 1740, the chair adjacent to a couch 1750. The person sitting in thechair is near the center of the photographed scene, while the TV 1740 ison the left-hand side of the photographed scene, and the couch 1750 ison the right-hand side of the photographed scene. In the first combinedimage 1710 (where P=0), the TV 1740 and the couch 1750 appear toostrongly horizontally squished together, curved, and/or slanted towardthe camera, and thus appear unnatural. In the third combined image 1730(where P=1), the TV 1740 and the couch 1750 appear stretched out to thesides away from the seated person, and appear unnaturally long andhorizontally-stretched relative to the other objects in the scene. Inthe second combined image 1720 (where P=0.8), the TV 1740 and the couch1750 appear to naturally reflect the photographed scene.

FIG. 18 is a conceptual diagram illustrating a graph 1800 comparingdifferent degrees of curvature of curved perspective distortioncorrection with respect to a flat perspective distortion. The differentdegrees of curvature of curved perspective distortion correction 1525are applied by mapping to a variable-curvature perspective-correctedimage plane using the equation

$x^{''} = \frac{f \cdot {\tan( {P \cdot \alpha} )}}{P}$

as discussed above. The graph 1800 is based on the equation

$x^{''} = {\frac{f \cdot {\tan( {P \cdot \alpha} )}}{P}.}$

The horizontal axis of the graph 1800 represents a normalized x withP=1, or the mapping output of the flat perspective correction with anangle range 0<=α<=65 degree, The vertical axis represents x″, or themapping outputs of the variable-curvature perspective correction withdifferent degrees of curvatures in the same scale as the horizontalaxis.

The graph 1800 illustrates five lines 1805, 1810, 1815, 1820, and 1825.The first line 1805 corresponds to P=0. The second line 1810 correspondsto P=0.4. The third line 1815 corresponds to P=0.6. The fourth line 1820corresponds to P=0.8. The fifth line 1825 corresponds to P=1.0.

FIG. 19 is a flow diagram illustrating an example process for performingcurved perspective distortion correction. In some examples, theoperations in the process 1300 may be performed by an imaging system. Insome examples, the imaging system is the device 500. In some examples,the imaging system includes at least one of the camera 112, the camera206, the device 500, the imaging architecture illustrated in conceptualdiagram 600, the imaging architecture illustrated in conceptual diagram700, the imaging architecture illustrated in conceptual diagram 800, theimaging architecture illustrated in conceptual diagram 900, the imagingarchitecture illustrated in conceptual diagram 1100, the imagingarchitecture illustrated in conceptual diagram 1200, the imagingarchitecture illustrated in conceptual diagram 1240, the imagingarchitecture illustrated in conceptual diagram 1260, the imagingarchitecture illustrated in conceptual diagram 1600, least one of animage capture and processing system 2000, an image capture device 2005A,an image processing device 2005B, an image processor 2050, a hostprocessor 2052, an ISP 2054, a computing system 2100, one or morenetwork servers of a cloud service, or a combination thereof.

At operation 1905, the imaging system receives a first image of a scenecaptured by a first image sensor of a first camera. The first imagecorresponds to a flat planar image plane. In some examples, the firstimage corresponds to the flat planar image plane because the first imagesensor corresponds to the flat planar image plane in shape and/orrelative dimensions. In some examples, the first image corresponds tothe flat planar image plane because the first image is projected ontothe flat planar image plane using flat perspective distortion correction1515.

At operation 1910, the imaging system identifies a curvedperspective-corrected image plane. In some examples, the imaging systemidentifies the curved perspective-corrected image plane to be the curvedperspective-corrected image plane 1630 of the diagram 1650 using theequation x′=f·α. In some examples, the imaging system imaging systemidentifies a curved perspective-corrected image plane to be avariable-curvature perspective-corrected image plane using the equation

$x^{''} = {\frac{f \cdot {\tan( {P \cdot \alpha} )}}{P}.}$

At operation 1915, the imaging system generates a perspective-correctedfirst image at least by projecting image data of the first image fromthe flat planar image plane corresponding to the first image sensor ontothe curved perspective-corrected image plane.

The process 1900 may be an example of the modification of the firstimage and/or the second image using perspective distortion of operation1365. In some examples, the first image received in operation 1905 maybe an example of the first image received in operation 1355, and theperspective-corrected first image of operation 1915 may be an example ofthe first image following the modifications using perspective distortionof operation 1365. In some examples, the first image received inoperation 1905 may be an example of the second image received inoperation 1360, and the perspective-corrected first image of operation1915 may be an example of the second image following the modificationsusing perspective distortion of operation 1365.

In some examples, P may be predetermined. In the imaging system mayreceive user inputs from a user through a user interface of the imagingsystem, and the imaging system can determine P based on the user inputs.In some examples, the imaging system may automatically determine P bydetecting that the scene appears warped in the first image, or is likelyto appear warped if a flat perspective distortion correction 1515 aloneis applied to the first image. In some examples, the imaging system mayautomatically determine P to fix or optimize the appearance of the scenein the first image when the imaging system determines that the sceneappears warped in the first image, or is likely to appear warped if aflat perspective distortion correction 1515 alone is applied to thefirst image. In some examples, the imaging system may automaticallydetermine P based on object distance, distribution, and surfaceorientation of objects and/or surfaces in the scene photographed in thefirst image. The imaging system may determine object distance,distribution, and/or surface orientation of objects and/or surfaces inthe scene based on object detection and/or recognition using the firstimage and/or one or more other images captured by the one or morecameras of the imaging system. For example, the imaging system can usefacial detection and/or facial recognition to identify human beings inthe scene, how close those human beings are to the camera (e.g., basedon the size of the face as determined via inter-eye distance or anothermeasurement between facial features), which direction the human beingsare facing, and so forth. The imaging system may determine objectdistance, distribution, and/or surface orientation of objects and/orsurfaces in the scene based on one or more point cloud of the scenegenerated using one or more range sensors of the imaging system, such asone or more light detection and ranging (LIDAR) sensors, one or moreradio detection and ranging (RADAR) sensors, one or more soundnavigation and ranging (SONAR) sensors, one or more sound detection andranging (SODAR) sensors, one or more time-of-flight (TOF) sensors, oneor more structured light (SL) sensors, or a combination thereof.

In some examples, the imaging system may automatically determine P tofix or optimize the appearance of human beings, faces, or anotherspecific type of object detected in the first image using objectdetection, object recognition, facial detection, or facial recognition.For example, the imaging system may determine that the first imageincludes a depiction of an office building. The imaging system mayexpect the office building to have a rectangular prism shape (e.g., abox). The imaging system may automatically determine P to make theoffice building appear as close to the rectangular prism shape aspossible in the perspective-corrected first image, and for example sothat the perspective-corrected first image removes or reduces any curvesin the edges of the office building that appear in the first image. Theimaging system may determine that the first image includes a depictionof a person's face. The imaging system may recognize the person's facebased on a comparison to other pre-stored images of the person's face,and can automatically determine P to make the person's face as depictedin the perspective-corrected first image appear as close as possible tothe pre-stored images of the person's face.

In some examples, the curved perspective distortion correction can beapplied only to a portion of the first image, rather than to theentirety of the first image. For example, in the combined image 1520depicting the five soda cans, the leftmost and rightmost soda cans inthe combined image 1520 appear most warped. The curved perspectivedistortion correction can, in some examples, be applied only to theregions of the combined image 1520 that include the depictions of theleftmost and rightmost soda cans.

In some examples, the curved perspective distortion correction can beapplied to reduce various types of distortion, including distortionbrought about by wide-angle lenses and/or fisheye lenses.

FIG. 20 is a block diagram illustrating an architecture of an imagecapture and processing system 2000. Each of the cameras, lenses, and/orimage sensors discussed with respect to previous figures may be includedin an image capture and processing system 2000. For example, the lens104 and image sensor 106 of FIG. 1 can be included in an image captureand processing system 2000. The camera 206 of FIG. 2 can be an exampleof an image capture and processing system 2000. The first camera 501 andthe second camera 502 of FIG. 5 can each be an example of an imagecapture and processing system 2000. The first camera lens 606 and thefirst image sensor 602 of FIG. 6 can be included in one image captureand processing system 2000, while the second camera lens 608 and thesecond image sensor 604 of FIG. 6 can be included in another imagecapture and processing system 2000. The camera lens 704 and the imagesensor 702 of FIG. 7 can be included in an image capture and processingsystem 2000. The first camera lens 806 and the first image sensor 802 ofFIG. 8 can be included in one image capture and processing system 2000,while the second camera lens 808 and the second image sensor 804 of FIG.8 can be included in another image capture and processing system 2000.The first camera lens 906 and the first image sensor 902 of FIG. 9 canbe included in one image capture and processing system 2000, while thesecond camera lens 908 and the second image sensor 904 of FIG. 9 can beincluded in another image capture and processing system 2000. The imagesensor 1004 of FIG. 10A can be included in an image capture andprocessing system 2000. The first camera and the second camera of FIG.10C can each be an example of an image capture and processing system2000. The first camera lens 1106 and the first image sensor 1102 of FIG.11 can be included in one image capture and processing system 2000,while the second camera lens 1108 and the second image sensor 1104 ofFIG. 11 can be included in another image capture and processing system2000. The first camera lens 1206 and the first image sensor 1202 ofFIGS. 12A-12C can be included in one image capture and processing system2000, while the second camera lens 1208 and the second image sensor 1204of FIGS. 12A-12B can be included in another image capture and processingsystem 2000. The first lens and the first image sensor mentioned in theflow chart of example operation 1300 of FIG. 13A can be included in oneimage capture and processing system 2000, while the second lens and thesecond image sensor mentioned in the flow chart of example operation1300 of FIG. 13A can be included in another image capture and processingsystem 2000. The first lens and the first image sensor mentioned in theflow chart of example operation 1300 of FIG. 13B can be included in oneimage capture and processing system 2000, while the second lens and thesecond image sensor mentioned in the flow chart of example operation1300 of FIG. 13B can be included in another image capture and processingsystem 2000. The first camera mentioned in the flow chart of exampleoperation 1400 of FIG. 14 can be included in one image capture andprocessing system 2000, while the second camera mentioned in the flowchart of example operation 1400 of FIG. 14 can be included in anotherimage capture and processing system 2000.

The image capture and processing system 2000 includes various componentsthat are used to capture and process images of scenes (e.g., an image ofa scene 2010). The image capture and processing system 2000 can capturestandalone images (or photographs) and/or can capture videos thatinclude multiple images (or video frames) in a particular sequence. Alens 2015 of the system 2000 faces a scene 2010 and receives light fromthe scene 2010. The lens 2015 bends the light toward the image sensor2030. The light received by the lens 2015 passes through an aperturecontrolled by one or more control mechanisms 2020 and is received by animage sensor 2030.

The one or more control mechanisms 2020 may control exposure, focus,and/or zoom based on information from the image sensor 2030 and/or basedon information from the image processor 2050. The one or more controlmechanisms 2020 may include multiple mechanisms and components; forinstance, the control mechanisms 2020 may include one or more exposurecontrol mechanisms 2025A, one or more focus control mechanisms 2025B,and/or one or more zoom control mechanisms 2025C. The one or morecontrol mechanisms 2020 may also include additional control mechanismsbesides those that are illustrated, such as control mechanismscontrolling analog gain, flash, HDR, depth of field, and/or other imagecapture properties.

The focus control mechanism 2025B of the control mechanisms 2020 canobtain a focus setting. In some examples, focus control mechanism 2025Bstore the focus setting in a memory register. Based on the focussetting, the focus control mechanism 2025B can adjust the position ofthe lens 2015 relative to the position of the image sensor 2030. Forexample, based on the focus setting, the focus control mechanism 2025Bcan move the lens 2015 closer to the image sensor 2030 or farther fromthe image sensor 2030 by actuating a motor or servo (or other lensmechanism), thereby adjusting focus. In some cases, additional lensesmay be included in the system 2000, such as one or more microlenses overeach photodiode of the image sensor 2030, which each bend the lightreceived from the lens 2015 toward the corresponding photodiode beforethe light reaches the photodiode. The focus setting may be determinedvia contrast detection autofocus (CDAF), phase detection autofocus(PDAF), hybrid autofocus (HAF), or some combination thereof. The focussetting may be determined using the control mechanism 2020, the imagesensor 2030, and/or the image processor 2050. The focus setting may bereferred to as an image capture setting and/or an image processingsetting.

The exposure control mechanism 2025A of the control mechanisms 2020 canobtain an exposure setting. In some cases, the exposure controlmechanism 2025A stores the exposure setting in a memory register. Basedon this exposure setting, the exposure control mechanism 2025A cancontrol a size of the aperture (e.g., aperture size or f/stop), aduration of time for which the aperture is open (e.g., exposure time orshutter speed), a sensitivity of the image sensor 2030 (e.g., ISO speedor film speed), analog gain applied by the image sensor 2030, or anycombination thereof. The exposure setting may be referred to as an imagecapture setting and/or an image processing setting.

The zoom control mechanism 2025C of the control mechanisms 2020 canobtain a zoom setting. In some examples, the zoom control mechanism2025C stores the zoom setting in a memory register. Based on the zoomsetting, the zoom control mechanism 2025C can control a focal length ofan assembly of lens elements (lens assembly) that includes the lens 2015and one or more additional lenses. For example, the zoom controlmechanism 2025C can control the focal length of the lens assembly byactuating one or more motors or servos (or other lens mechanism) to moveone or more of the lenses relative to one another. The zoom setting maybe referred to as an image capture setting and/or an image processingsetting. In some examples, the lens assembly may include a parfocal zoomlens or a varifocal zoom lens. In some examples, the lens assembly mayinclude a focusing lens (which can be lens 2015 in some cases) thatreceives the light from the scene 2010 first, with the light thenpassing through an afocal zoom system between the focusing lens (e.g.,lens 2015) and the image sensor 2030 before the light reaches the imagesensor 2030. The afocal zoom system may, in some cases, include twopositive (e.g., converging, convex) lenses of equal or similar focallength (e.g., within a threshold difference of one another) with anegative (e.g., diverging, concave) lens between them. In some cases,the zoom control mechanism 2025C moves one or more of the lenses in theafocal zoom system, such as the negative lens and one or both of thepositive lenses.

The image sensor 2030 includes one or more arrays of photodiodes orother photosensitive elements. Each photodiode measures an amount oflight that eventually corresponds to a particular pixel in the imageproduced by the image sensor 2030. In some cases, different photodiodesmay be covered by different color filters, and may thus measure lightmatching the color of the filter covering the photodiode. For instance,Bayer color filters include red color filters, blue color filters, andgreen color filters, with each pixel of the image generated based on redlight data from at least one photodiode covered in a red color filter,blue light data from at least one photodiode covered in a blue colorfilter, and green light data from at least one photodiode covered in agreen color filter. Other types of color filters may use yellow,magenta, and/or cyan (also referred to as “emerald”) color filtersinstead of or in addition to red, blue, and/or green color filters. Someimage sensors (e.g., image sensor 2030) may lack color filtersaltogether, and may instead use different photodiodes throughout thepixel array (in some cases vertically stacked). The differentphotodiodes throughout the pixel array can have different spectralsensitivity curves, therefore responding to different wavelengths oflight. Monochrome image sensors may also lack color filters andtherefore lack color depth.

In some cases, the image sensor 2030 may alternately or additionallyinclude opaque and/or reflective masks that block light from reachingcertain photodiodes, or portions of certain photodiodes, at certaintimes and/or from certain angles, which may be used for phase detectionautofocus (PDAF). The image sensor 2030 may also include an analog gainamplifier to amplify the analog signals output by the photodiodes and/oran analog to digital converter (ADC) to convert the analog signalsoutput of the photodiodes (and/or amplified by the analog gainamplifier) into digital signals. In some cases, certain components orfunctions discussed with respect to one or more of the controlmechanisms 2020 may be included instead or additionally in the imagesensor 2030. The image sensor 2030 may be a charge-coupled device (CCD)sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixelsensor (APS), a complimentary metal-oxide semiconductor (CMOS), anN-type metal-oxide semiconductor (NMOS), a hybrid CCD/CMOS sensor (e.g.,sCMOS), or some other combination thereof.

The image processor 2050 may include one or more processors, such as oneor more image signal processors (ISPs) (including ISP 2054), one or morehost processors (including host processor 2052), and/or one or more ofany other type of processor 2110 discussed with respect to theprocessing system 2100. The host processor 2052 can be a digital signalprocessor (DSP) and/or other type of processor. In some implementations,the image processor 2050 is a single integrated circuit or chip (e.g.,referred to as a system-on-chip or SoC) that includes the host processor2052 and the ISP 2054. In some cases, the chip can also include one ormore input/output ports (e.g., input/output (I/O) ports 2056), centralprocessing units (CPUs), graphics processing units (GPUs), broadbandmodems (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components(e.g., Bluetooth™, Global Positioning System (GPS), etc.), anycombination thereof, and/or other components. The I/O ports 2056 caninclude any suitable input/output ports or interface according to one ormore protocol or specification, such as an Inter-Integrated Circuit 2(I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a SerialPeripheral Interface (SPI) interface, a serial General PurposeInput/Output (GPIO) interface, a Mobile Industry Processor Interface(MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, anAdvanced High-performance Bus (AHB) bus, any combination thereof, and/orother input/output port. In one illustrative example, the host processor2052 can communicate with the image sensor 2030 using an I2C port, andthe ISP 2054 can communicate with the image sensor 2030 using an MIPIport.

The image processor 2050 may perform a number of tasks, such asde-mosaicing, color space conversion, image frame downsampling, pixelinterpolation, automatic exposure (AE) control, automatic gain control(AGC), CDAF, PDAF, automatic white balance, merging of image frames toform an HDR image, image recognition, object recognition, featurerecognition, receipt of inputs, managing outputs, managing memory, orsome combination thereof. The image processor 2050 may store imageframes and/or processed images in random access memory (RAM) 2040/2020,read-only memory (ROM) 2045/2025, a cache, a memory unit, anotherstorage device, or some combination thereof.

Various input/output (I/O) devices 2060 may be connected to the imageprocessor 2050. The I/O devices 2060 can include a display screen, akeyboard, a keypad, a touchscreen, a trackpad, a touch-sensitivesurface, a printer, any other output devices 2135, any other inputdevices 2145, or some combination thereof. In some cases, a caption maybe input into the image processing device 2005B through a physicalkeyboard or keypad of the I/O devices 2060, or through a virtualkeyboard or keypad of a touchscreen of the I/O devices 2060. The I/O2060 may include one or more ports, jacks, or other connectors thatenable a wired connection between the system 2000 and one or moreperipheral devices, over which the system 2000 may receive data from theone or more peripheral device and/or transmit data to the one or moreperipheral devices. The I/O 2060 may include one or more wirelesstransceivers that enable a wireless connection between the system 2000and one or more peripheral devices, over which the system 2000 mayreceive data from the one or more peripheral device and/or transmit datato the one or more peripheral devices. The peripheral devices mayinclude any of the previously-discussed types of I/O devices 2060 andmay themselves be considered I/O devices 2060 once they are coupled tothe ports, jacks, wireless transceivers, or other wired and/or wirelessconnectors.

In some cases, the image capture and processing system 2000 may be asingle device. In some cases, the image capture and processing system2000 may be two or more separate devices, including an image capturedevice 2005A (e.g., a camera) and an image processing device 2005B(e.g., a computing device coupled to the camera). In someimplementations, the image capture device 2005A and the image processingdevice 2005B may be coupled together, for example via one or more wires,cables, or other electrical connectors, and/or wirelessly via one ormore wireless transceivers. In some implementations, the image capturedevice 2005A and the image processing device 2005B may be disconnectedfrom one another.

As shown in FIG. 20, a vertical dashed line divides the image captureand processing system 2000 of FIG. 20 into two portions that representthe image capture device 2005A and the image processing device 2005B,respectively. The image capture device 2005A includes the lens 2015,control mechanisms 2020, and the image sensor 2030. The image processingdevice 2005B includes the image processor 2050 (including the ISP 2054and the host processor 2052), the RAM 2040, the ROM 2045, and the I/O2060. In some cases, certain components illustrated in the image capturedevice 2005A, such as the ISP 2054 and/or the host processor 2052, maybe included in the image capture device 2005A.

The image capture and processing system 2000 can include an electronicdevice, such as a mobile or stationary telephone handset (e.g.,smartphone, cellular telephone, or the like), a desktop computer, alaptop or notebook computer, a tablet computer, a set-top box, atelevision, a camera, a display device, a digital media player, a videogaming console, a video streaming device, an Internet Protocol (IP)camera, or any other suitable electronic device. In some examples, theimage capture and processing system 2000 can include one or morewireless transceivers for wireless communications, such as cellularnetwork communications, 802.11 wi-fi communications, wireless local areanetwork (WLAN) communications, or some combination thereof. In someimplementations, the image capture device 2005A and the image processingdevice 2005B can be different devices. For instance, the image capturedevice 2005A can include a camera device and the image processing device2005B can include a computing device, such as a mobile handset, adesktop computer, or other computing device.

While the image capture and processing system 2000 is shown to includecertain components, one of ordinary skill will appreciate that the imagecapture and processing system 2000 can include more components thanthose shown in FIG. 20. The components of the image capture andprocessing system 2000 can include software, hardware, or one or morecombinations of software and hardware. For example, in someimplementations, the components of the image capture and processingsystem 2000 can include and/or can be implemented using electroniccircuits or other electronic hardware, which can include one or moreprogrammable electronic circuits (e.g., microprocessors, GPUs, DSPs,CPUs, and/or other suitable electronic circuits), and/or can includeand/or be implemented using computer software, firmware, or anycombination thereof, to perform the various operations described herein.The software and/or firmware can include one or more instructions storedon a computer-readable storage medium and executable by one or moreprocessors of the electronic device implementing the image capture andprocessing system 2000.

FIG. 21 is a diagram illustrating an example of a system forimplementing certain aspects of the present technology. In particular,FIG. 21 illustrates an example of computing system 2100, which can befor example any computing device making up internal computing system, aremote computing system, a camera, or any component thereof in which thecomponents of the system are in communication with each other usingconnection 2105. Connection 2105 can be a physical connection using abus, or a direct connection into processor 2110, such as in a chipsetarchitecture. Connection 2105 can also be a virtual connection,networked connection, or logical connection.

In some embodiments, computing system 2100 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example system 2100 includes at least one processing unit (CPU orprocessor) 2110 and connection 2105 that couples various systemcomponents including system memory 2115, such as read-only memory (ROM)2120 and random access memory (RAM) 2125 to processor 2110. Computingsystem 2100 can include a cache 2112 of high-speed memory connecteddirectly with, in close proximity to, or integrated as part of processor2110.

Processor 2110 can include any general purpose processor and a hardwareservice or software service, such as services 2132, 2134, and 2136stored in storage device 2130, configured to control processor 2110 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. Processor 2110 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction, computing system 2100 includes an inputdevice 2145, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 2100 can also include output device 2135, which can be one ormore of a number of output mechanisms. In some instances, multimodalsystems can enable a user to provide multiple types of input/output tocommunicate with computing system 2100. Computing system 2100 caninclude communications interface 2140, which can generally govern andmanage the user input and system output. The communication interface mayperform or facilitate receipt and/or transmission wired or wirelesscommunications using wired and/or wireless transceivers, including thosemaking use of an audio jack/plug, a microphone jack/plug, a universalserial bus (USB) port/plug, an Apple® Lightning® port/plug, an Ethernetport/plug, a fiber optic port/plug, a proprietary wired port/plug, aBLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE)wireless signal transfer, an IBEACON® wireless signal transfer, aradio-frequency identification (RFID) wireless signal transfer,near-field communications (NFC) wireless signal transfer, dedicatedshort range communication (DSRC) wireless signal transfer, 802.11 Wi-Fiwireless signal transfer, wireless local area network (WLAN) signaltransfer, Visible Light Communication (VLC), Worldwide Interoperabilityfor Microwave Access (WiMAX), Infrared (IR) communication wirelesssignal transfer, Public Switched Telephone Network (PSTN) signaltransfer, Integrated Services Digital Network (ISDN) signal transfer,3G/4G/5G/LTE cellular data network wireless signal transfer, ad-hocnetwork signal transfer, radio wave signal transfer, microwave signaltransfer, infrared signal transfer, visible light signal transfer,ultraviolet light signal transfer, wireless signal transfer along theelectromagnetic spectrum, or some combination thereof. Thecommunications interface 2140 may also include one or more GlobalNavigation Satellite System (GNSS) receivers or transceivers that areused to determine a location of the computing system 2100 based onreceipt of one or more signals from one or more satellites associatedwith one or more GNSS systems. GNSS systems include, but are not limitedto, the US-based Global Positioning System (GPS), the Russia-basedGlobal Navigation Satellite System (GLONASS), the China-based BeiDouNavigation Satellite System (BDS), and the Europe-based Galileo GNSS.There is no restriction on operating on any particular hardwarearrangement, and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 2130 can be a non-volatile and/or non-transitory and/orcomputer-readable memory device and can be a hard disk or other types ofcomputer readable media which can store data that are accessible by acomputer, such as magnetic cassettes, flash memory cards, solid statememory devices, digital versatile disks, cartridges, a floppy disk, aflexible disk, a hard disk, magnetic tape, a magnetic strip/stripe, anyother magnetic storage medium, flash memory, memristor memory, any othersolid-state memory, a compact disc read only memory (CD-ROM) opticaldisc, a rewritable compact disc (CD) optical disc, digital video disk(DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographicoptical disk, another optical medium, a secure digital (SD) card, amicro secure digital (microSD) card, a Memory Stick® card, a smartcardchip, a EMV chip, a subscriber identity module (SIM) card, amini/micro/nano/pico SIM card, another integrated circuit (IC)chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM(DRAM), read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cachememory (L1/L2/L3/L4/L5/L #), resistive random-access memory(RRAM/ReRAM), phase change memory (PCM), spin transfer torque RAM(STT-RAM), another memory chip or cartridge, and/or a combinationthereof.

The storage device 2130 can include software services, servers,services, etc., that when the code that defines such software isexecuted by the processor 2110, it causes the system to perform afunction. In some embodiments, a hardware service that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 2110, connection 2105, output device 2135,etc., to carry out the function.

As used herein, the term “computer-readable medium” includes, but is notlimited to, portable or non-portable storage devices, optical storagedevices, and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A computer-readable medium mayinclude a non-transitory medium in which data can be stored and thatdoes not include carrier waves and/or transitory electronic signalspropagating wirelessly or over wired connections. Examples of anon-transitory medium may include, but are not limited to, a magneticdisk or tape, optical storage media such as compact disk (CD) or digitalversatile disk (DVD), flash memory, memory or memory devices. Acomputer-readable medium may have stored thereon code and/ormachine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted using any suitable means including memory sharing,message passing, token passing, network transmission, or the like.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Specific details are provided in the description above to provide athorough understanding of the embodiments and examples provided herein.However, it will be understood by one of ordinary skill in the art thatthe embodiments may be practiced without these specific details. Forclarity of explanation, in some instances the present technology may bepresented as including individual functional blocks including functionalblocks comprising devices, device components, steps or routines in amethod embodied in software, or combinations of hardware and software.Additional components may be used other than those shown in the figuresand/or described herein. For example, circuits, systems, networks,processes, and other components may be shown as components in blockdiagram form in order not to obscure the embodiments in unnecessarydetail. In other instances, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Individual embodiments may be described above as a process or methodwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel, concurrently, or contemporaneously. In addition,the order of the operations may be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process may correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

Processes and methods according to the above-described examples can beimplemented using computer-executable instructions that are stored orotherwise available from computer-readable media. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general purpose computer, special purpose computer, or aprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware,source code, etc. Examples of computer-readable media that may be usedto store instructions, information used, and/or information createdduring methods according to described examples include magnetic oroptical disks, flash memory, USB devices provided with non-volatilememory, networked storage devices, and so on.

Devices implementing processes and methods according to thesedisclosures can include hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof,and can take any of a variety of form factors. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks (e.g., a computer-programproduct) may be stored in a computer-readable or machine-readablemedium. A processor(s) may perform the necessary tasks. Typical examplesof form factors include laptops, smart phones, mobile phones, tabletdevices or other small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are example means for providing the functionsdescribed in the disclosure.

In the foregoing description, aspects of the application are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the application is not limited thereto. Thus,while illustrative embodiments of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art. Various features and aspects of theabove-described application may be used individually or jointly.Further, embodiments can be utilized in any number of environments andapplications beyond those described herein without departing from thebroader spirit and scope of the specification. The specification anddrawings are, accordingly, to be regarded as illustrative rather thanrestrictive. For the purposes of illustration, methods were described ina particular order. It should be appreciated that in alternateembodiments, the methods may be performed in a different order than thatdescribed.

One of ordinary skill will appreciate that the less than (“<”) andgreater than (“>”) symbols or terminology used herein can be replacedwith less than or equal to (“≤”) and greater than or equal to (“≥”)symbols, respectively, without departing from the scope of thisdescription.

Where components are described as being “configured to” perform certainoperations, such configuration can be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

The phrase “coupled to” refers to any component that is physicallyconnected to another component either directly or indirectly, and/or anycomponent that is in communication with another component (e.g.,connected to the other component over a wired or wireless connection,and/or other suitable communication interface) either directly orindirectly.

Claim language or other language reciting “at least one of” a set and/or“one or more” of a set indicates that one member of the set or multiplemembers of the set (in any combination) satisfy the claim. For example,claim language reciting “at least one of A and B” means A, B, or A andB. In another example, claim language reciting “at least one of A, B,and C” means A, B, C, or A and B, or A and C, or B and C, or A and B andC. The language “at least one of” a set and/or “one or more” of a setdoes not limit the set to the items listed in the set. For example,claim language reciting “at least one of A and B” can mean A, B, or Aand B, and can additionally include items not listed in the set of A andB.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software,firmware, or combinations thereof. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present application.

The techniques described herein may also be implemented in electronichardware, computer software, firmware, or any combination thereof. Suchtechniques may be implemented in any of a variety of devices such asgeneral purposes computers, wireless communication device handsets, orintegrated circuit devices having multiple uses including application inwireless communication device handsets and other devices. Any featuresdescribed as modules or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a computer-readable data storage mediumcomprising program code including instructions that, when executed,performs one or more of the methods described above. Thecomputer-readable data storage medium may form part of a computerprogram product, which may include packaging materials. Thecomputer-readable medium may comprise memory or data storage media, suchas random access memory (RAM) such as synchronous dynamic random accessmemory (SDRAM), read-only memory (ROM), non-volatile random accessmemory (NVRAM), electrically erasable programmable read-only memory(EEPROM), FLASH memory, magnetic or optical data storage media, and thelike. The techniques additionally, or alternatively, may be realized atleast in part by a computer-readable communication medium that carriesor communicates program code in the form of instructions or datastructures and that can be accessed, read, and/or executed by acomputer, such as propagated signals or waves.

The program code may be executed by a processor, which may include oneor more processors, such as one or more digital signal processors(DSPs), general purpose microprocessors, an application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), orother equivalent integrated or discrete logic circuitry. Such aprocessor may be configured to perform any of the techniques describedin this disclosure. A general purpose processor may be a microprocessor;but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Accordingly, the term “processor,” as used herein mayrefer to any of the foregoing structure, any combination of theforegoing structure, or any other structure or apparatus suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated software modules or hardware modules configured for encodingand decoding, or incorporated in a combined video encoder-decoder(CODEC).

As noted above, while the present disclosure shows illustrative aspects,it should be noted that various changes and modifications could be madeherein without departing from the scope of the appended claims.Additionally, the functions, steps or actions of the method claims inaccordance with aspects described herein need not be performed in anyparticular order unless expressly stated otherwise. Furthermore,although elements may be described or claimed in the singular, theplural is contemplated unless limitation to the singular is explicitlystated. Accordingly, the disclosure is not limited to the illustratedexamples and any means for performing the functionality described hereinare included in aspects of the disclosure.

Illustrative aspects of the disclosure include:

Aspect 1: A device for digital imaging, comprising: a memory; and one ormore processors configured to: receive a first image frame of a scenecaptured by a first camera having a first entrance pupil, wherein: afirst light redirection element redirects a first light towards thefirst camera; and the first camera captures the first image frame basedon the first light redirected by the first light redirection elementtowards the first camera; receive a second image frame of the scenecaptured by a second camera having a second entrance pupil, wherein: asecond light redirection element redirects a second light towards thesecond camera; the second camera captures the second image frame basedon the second light redirected by the second light redirection elementtowards the second camera; and a first center of the first camera lensentrance pupil and a second center of the second camera lens entrancepupil virtually overlap; and generate a combined image from the firstimage frame and the second image frame, wherein the combined imageincludes a first field of view that is wider than a field of view of thefirst image frame or a field of view of the second image frame.

Aspect 2: A device according to Aspect 1, wherein the one or moreprocessors are further configured to: adjust the first image frame froma first perspective to a common perspective; and adjust the second imageframe from a second perspective to the common perspective, wherein thecommon perspective is between the first perspective and the secondperspective.

Aspect 3: A device according to any one of Aspects 1 or 2, wherein theone or more processors are configured to: align and stitch the firstimage frame and the second image frame together to generate the combinedimage, wherein aligning the image frames is based on scene matching inan overlapping portion of the scene in the first image frame and in thesecond image frame.

Aspect 4: A device according to any one of Aspects 1 to 3, furthercomprising: the first camera; the second camera; the first lightredirection element; and the second light redirection element, whereinthe first camera lens entrance pupil center and the second camera lensentrance pupil center virtually overlap at a first location without thefirst camera lens and the second camera lens physically overlapping atthe first location based on positions of the first camera, the secondcamera, and the first and second light redirection elements.

Aspect 5: A device according to Aspect 4, wherein: the first lightredirection element includes a first mirror configured to reflect thefirst light towards the first camera; and the second light redirectionelement includes a second mirror configured to reflect the second lighttowards the second camera.

Aspect 6: A device according to Aspect 5, wherein: the first lightredirection element also includes a first prism configured to refractthe first light; and the second light redirection element also includesa second prism configured to refract the second light.

Aspect 7: A device according to Aspect 6, wherein: the first mirror ison a first side of the first prism; the second mirror is on a secondside of the second prism; one or more corners of the first prism areprevented from reflecting light from a first camera lens surface of thefirst camera back towards the first camera lens; and one or more cornersof the second prism are prevented from reflecting light from a secondcamera lens surface of the second camera back towards the second cameralens.

Aspect 8: A device according to Aspect 7, wherein: the one or morecorners of the first prism include a chamfered edge with a lightabsorbing coating applied to the chamfered edge of the first prism; andthe one or more corners of the second prism include a chamfered edgewith a light absorbing coating applied to the chamfered edge of thesecond prism.

Aspect 9: A device according to any one of Aspects 1 to 8, wherein: thefirst light redirection element and the second light redirection elementare a single redirection element; the single redirection elementincludes the first prism and the second prism; the first mirror is on afirst side of the first prism; the second mirror is on a second side ofthe second prism; and the first prism and the second prism overlap andconnect at the first side and the second side in the single redirectionelement.

Aspect 10: A device according to any one of Aspects 1 to 8, wherein thefirst image frame and the second image frame are captured concurrently.

Aspect 11: A method for digital imaging, comprising: receiving a firstimage frame of a scene captured by a first camera having a firstentrance pupil, wherein: a first light redirection element redirects afirst light towards the first camera; and the first camera captures thefirst image frame based on the first light redirected by the first lightredirection element towards the first camera; receiving a second imageframe of the scene captured by a second camera having a second entrancepupil, wherein: a second light redirection element redirects a secondlight towards the second camera; the second camera captures the secondimage frame based on the second light redirected by the second lightredirection element towards the second camera; and a first center of thefirst camera lens entrance pupil and a second center of the secondcamera lens entrance pupil virtually overlap; and generating a combinedimage from the first image frame and the second image frame, wherein thecombined image includes a first field of view that is wider than a fieldof view of the first image frame or a field of view of the second imageframe.

Aspect 12: A method according to Aspect 11, further comprising:adjusting the first image frame from a first perspective to a commonperspective; and adjusting the second image frame from a secondperspective to the common perspective, wherein the common perspective isbetween the first perspective and the second perspective.

Aspect 13: A method according to any one of Aspects 11 or 12, furthercomprising: aligning and stitching the first image frame and the secondimage frame together to generate the combined image, wherein aligningthe image frames is based on scene matching in an overlapping portion ofthe scene in the first image frame and in the second image frame.

Aspect 14: A method according to any one of Aspects 11 to 13, furthercomprising: redirecting the first light by the first light redirectionelement; redirecting the second light by the second light redirectionelement; capturing the first image frame by the first camera; andcapturing the second image frame by the second camera, wherein the firstcamera lens entrance pupil center and the second camera lens entrancepupil center virtually overlap at a first location without the firstcamera lens and the second camera lens physically overlapping at thefirst location based on positions of the first camera, the secondcamera, and the first and second light redirection elements.

Aspect 15: A method according to Aspect 14, wherein: redirecting thefirst light by the first light redirection element includes reflectingthe first light by a first mirror towards the first camera; andredirecting the second light by the second light redirection elementincludes reflecting the second light by a second mirror towards thesecond camera.

Aspect 16: A method according to Aspect 15, wherein: redirecting thefirst light by the first light redirection element also includesrefracting the first light by a first prism; and redirecting the secondlight by the second light redirection element also includes refractingthe second light by a second prism.

Aspect 17: A method according to Aspect 16, wherein: the first mirror ison a first side of the first prism; the second mirror is on a secondside of the second prism; one or more corners of the first prism areprevented from reflecting light from a first camera lens surface of thefirst camera back towards the first camera lens; and one or more cornersof the second prism are prevented from reflecting light from a secondcamera lens surface of the second camera back towards the second cameralens.

Aspect 18: A method according to Aspect 17, wherein: the one or morecorners of the first prism include a chamfered edge with a lightabsorbing coating applied to the chamfered edge of the first prism; andthe one or more corners of the second prism include a chamfered edgewith a light absorbing coating applied to the chamfered edge of thesecond prism.

Aspect 19: A method according to any one of Aspects 11 to 18, wherein:the first light redirection element and the second light redirectionelement are a single redirection element; the single redirection elementincludes the first prism and the second prism; the first mirror is on afirst side of the first prism; the second mirror is on a second side ofthe second prism; and the first prism and the second prism overlap andconnect at the first side and the second side in the single redirectionelement.

Aspect 20: A method according to any one of Aspects 11 to 19, whereinthe first image frame and the second image frame are capturedconcurrently.

Aspect 21: A non-transitory, computer readable medium storinginstructions that, when executed by one or more processors of a devicefor digital imaging, causes the device to: receive a first image frameof a scene captured by a first camera having a first entrance pupil,wherein: a first light redirection element redirects a first lighttowards the first camera; and the first camera captures the first imageframe based on the first light redirected by the first light redirectionelement towards the first camera; receive a second image frame of thescene captured by a second camera having a second entrance pupil,wherein: a second light redirection element redirects a second lighttowards the second camera; the second camera captures the second imageframe based on the second light redirected by the second lightredirection element towards the second camera; and a first center of thefirst camera lens entrance pupil and a second center of the secondcamera lens entrance pupil virtually overlap; and generate a combinedimage from the first image frame and the second image frame, wherein thecombined image includes a first field of view that is wider than a fieldof view of the first image frame or a field of view of the second imageframe.

Aspect 22: A computer readable medium according to Aspect 21, whereinexecution of the instructions further causes the device to: adjust thefirst image frame from a first perspective to a common perspective; andadjust the second image frame from a second perspective to the commonperspective, wherein the common perspective is between the firstperspective and the second perspective.

Aspect 23: A computer readable medium according to any one of Aspects 21or 22, wherein execution of the instructions further causes the deviceto: align and stitch the first image frame and the second image frametogether to generate the combined image, wherein aligning the imageframes is based on scene matching in an overlapping portion of the scenein the first image frame and in the second image frame.

Aspect 24: A computer readable medium according to any one of Aspects 21to 23, wherein execution of the instructions further causes the deviceto: redirect the first light by the first light redirection element;redirect the second light by the second light redirection element;capture the first image frame by the first camera; and capture thesecond image frame by the second camera, wherein the first camera lensentrance pupil center and the second camera lens entrance pupil centervirtually overlap at a first location without the first camera lens andthe second camera lens physically overlapping at the first locationbased on positions of the first camera, the second camera, and the firstand second light redirection elements.

Aspect 25: A computer readable medium according to Aspect 24, wherein:redirecting the first light by the first light redirection elementincludes reflecting the first light by a first mirror towards the firstcamera; and redirecting the second light by the second light redirectionelement includes reflecting the second light by a second mirror towardsthe second camera.

Aspect 26: A computer readable medium according to Aspect 25, wherein:redirecting the first light by the first light redirection element alsoincludes refracting the first light by a first prism; and redirectingthe second light by the second light redirection element also includesrefracting the second light by a second prism.

Aspect 27: A computer readable medium according to Aspect 26, wherein:the first mirror is on a first side of the first prism; the secondmirror is on a second side of the second prism; one or more corners ofthe first prism are prevented from reflecting light from a first cameralens surface of the first camera back towards the first camera lens; andone or more corners of the second prism are prevented from reflectinglight from a second camera lens surface of the second camera backtowards the second camera lens.

Aspect 28: A computer readable medium according to Aspect 27, wherein:the one or more corners of the first prism include a chamfered edge witha light absorbing coating applied to the chamfered edge of the firstprism; and the one or more corners of the second prism include achamfered edge with a light absorbing coating applied to the chamferededge of the second prism.

Aspect 29: A computer readable medium according to any one of Aspects 21to 28, wherein: the first light redirection element and the second lightredirection element are a single redirection element; the singleredirection element includes the first prism and the second prism; thefirst mirror is on a first side of the first prism; the second mirror ison a second side of the second prism; and the first prism and the secondprism overlap and connect at the first side and the second side in thesingle redirection element.

Aspect 30: A computer readable medium according to any one of Aspects 21to 28, wherein the first image frame and the second image frame arecaptured concurrently.

Aspect 31: An apparatus for digital imaging, the apparatus comprising: amemory; and one or more processors configured to: receive a first imageof a scene captured by a first image sensor, wherein a first lightredirection element is configured to redirect a first light from a firstpath to a redirected first path toward the first image sensor, whereinthe first image sensor is configured to capture the first image based onreceipt of the first light at the first image sensor; receive a secondimage of the scene captured by a second image sensor, wherein a secondlight redirection element is configured to redirect a second light froma second path to a redirected second path toward the second imagesensor, wherein the second image sensor is configured to capture thesecond image based on receipt of the second light at the second imagesensor; modify at least one of the first image and the second imageusing a perspective distortion correction; and generate a combined imagefrom the first image and the second image in response to modification ofthe at least one of the first image and the second image using theperspective distortion correction, wherein the combined image includes acombined image field of view that is larger than at least one of a firstfield of view of the first image and a second field of view of thesecond image.

Aspect 32: An apparatus according to Aspect 31, wherein a virtualextension of the first path beyond the first light redirection elementintersects with a virtual extension of the second path intersect beyondthe second light redirection element.

Aspect 33: An apparatus according to any one of Aspects 31 or 32,wherein, to modify at least one of the first image and the second imageusing the perspective distortion correction, the one or more processorsare configured to: modify the first image from depicting a firstperspective to depicting a common perspective using the perspectivedistortion correction; and modify the second image from depicting asecond perspective to depicting the common perspective using theperspective distortion correction, wherein the common perspective isbetween the first perspective and the second perspective.

Aspect 34: An apparatus according to any one of Aspects 31 to 33,wherein, to modify at least one of the first image and the second imageusing the perspective distortion correction, the one or more processorsare configured to: identify depictions of one or more objects in imagedata of at least one of the first image and the second image; and modifythe image data at least in part by projecting the image data based onthe depictions of the one or more objects.

Aspect 35: An apparatus according to any one of Aspects 31 to 34,wherein, to generate the combined image from the first image and thesecond image, the one or more processors are configured to: align afirst portion of the first image with a second portion of the secondimage; and stitch the first image and the second image together based onthe first portion of the first image and the second portion of thesecond image being aligned.

Aspect 36: An apparatus according to any one of Aspects 31 to 35,further comprising: the first image sensor; the second image sensor; thefirst light redirection element; and the second light redirectionelement.

Aspect 37: An apparatus according to any one of Aspects 31 to 36,wherein: the first light redirection element includes a first reflectivesurface, wherein, to redirect the first light toward the first imagesensor, the first light redirection element uses the first reflectivesurface to reflect the first light toward the first image sensor; andthe second light redirection element includes a second reflectivesurface, wherein, to redirect the second light toward the second imagesensor, second light redirection element uses the second reflectivesurface to reflect the second light toward the second image sensor.

Aspect 38: An apparatus according to any one of Aspects 31 to 37,wherein: the first light redirection element includes a first prismconfigured to refract the first light; and the second light redirectionelement includes a second prism configured to refract the second light.

Aspect 39: An apparatus according to Aspect 38, wherein the first prismand the second prism are contiguous.

Aspect 40: An apparatus according to any one of Aspects 38 or 39,wherein a bridge joins a first edge of the first prism and a second edgeof the second prism, wherein the bridge is configured to preventreflection of light from at least one of first edge of the first prismand the second edge of the second prism.

Aspect 41: An apparatus according to any one of Aspects 38 to 40,wherein the first prism includes at least one chamfered edge, andwherein the second prism includes at least one chamfered edge.

Aspect 42: An apparatus according to any one of Aspects 38 to 41,wherein the first prism includes at least one edge with alight-absorbing coating, wherein the second prism includes at least oneedge with the light-absorbing coating.

Aspect 43: An apparatus according to any one of Aspects 38 to 42,wherein the first path is a path of the first light before the firstlight enters the first prism, wherein the second path is a path of thesecond light before the second light enters the second prism.

Aspect 44: An apparatus according to any one of Aspects 38 to 43,wherein the first prism includes a first reflective surface configuredto reflect the first light, wherein the second prism includes a secondreflective surface configured to reflect the second light.

Aspect 45: An apparatus according to any one of Aspects 38 to 44,wherein the first path is a path of the first light after the firstlight enters the first prism but before the first reflective surfacereflects the first light, wherein the second path is a path of thesecond light after the second light enters the second prism but beforethe second reflective surface reflects the second light.

Aspect 46: An apparatus according to any one of Aspects 31 to 45,wherein the first image and the second image are capturedcontemporaneously.

Aspect 47: An apparatus according to any one of Aspects 31 to 46,wherein the first light redirection element is fixed relative to thefirst image sensor, wherein the second light redirection element isfixed relative to the second image sensor.

Aspect 48: An apparatus according to any one of Aspects 31 to 47,wherein a first planar surface of the first image sensor faces a firstdirection, wherein a second planar surface of the second image sensorfaces a second direction that is parallel to the first direction.

Aspect 49: An apparatus according to any one of Aspects 31 to 48,wherein the one or more processors are configured to: modify at leastone of the first image and the second image using a brightnessuniformity correction.

Aspect 50: A method for digital imaging, the method comprising:receiving a first image of a scene captured by a first image sensor,wherein a first light redirection element redirects a first light from afirst path to a redirected first path toward the first image sensor,wherein the first image sensor captures the first image based on receiptof the first light at the first image sensor; receiving a second imageof the scene captured by a second image sensor, wherein a second lightredirection element redirects a second light from a second path to aredirected second path toward the second image sensor, wherein thesecond image sensor captures the second image based on receipt of thesecond light at the second image sensor; modifying at least one of thefirst image and the second image using a perspective distortioncorrection; and generating a combined image from the first image and thesecond image in response to modification of the at least one of thefirst image and the second image using the perspective distortioncorrection, wherein the combined image includes a combined image fieldof view that is larger than at least one of a first field of view of thefirst image and a second field of view of the second image.

Aspect 51: A method according to Aspect 50, wherein a virtual extensionof the first path beyond the first light redirection element intersectswith a virtual extension of the second path intersect beyond the secondlight redirection element.

Aspect 52: A method according to any one of Aspects 50 or 51, whereinmodifying at least one of the first image and the second image using theperspective distortion correction includes: modifying the first imagefrom depicting a first perspective to depicting a common perspectiveusing the perspective distortion correction; and modifying the secondimage from depicting a second perspective to depicting the commonperspective using the perspective distortion correction, wherein thecommon perspective is between the first perspective and the secondperspective.

Aspect 53: A method according to any one of Aspects 50 to 52, whereinmodifying at least one of the first image and the second image using theperspective distortion correction includes: identifying depictions ofone or more objects in image data of at least one of the first image andthe second image; and modifying the image data by projecting the imagedata based on the depictions of the one or more objects.

Aspect 54: A method according to any one of Aspects 50 to 53, wherein:the first light redirection element includes a first reflective surface,wherein, to redirect the first light toward the first image sensor, thefirst light redirection element uses the first reflective surface toreflect the first light toward the first image sensor; and the secondlight redirection element includes a second reflective surface, wherein,to redirect the second light toward the second image sensor, secondlight redirection element uses the second reflective surface to reflectthe second light toward the second image sensor.

Aspect 55: A method according to any one of Aspects 50 to 54, wherein:the first light redirection element includes a first prism configured torefract the first light; and the second light redirection elementincludes a second prism configured to refract the second light.

Aspect 56: A method according to Aspect 55, wherein the first prism andthe second prism are contiguous.

Aspect 57: A method according to any one of Aspects 55 or 56, wherein abridge joins a first edge of the first prism and a second edge of thesecond prism, wherein the bridge is configured to prevent reflection oflight from at least one of first edge of the first prism and the secondedge of the second prism.

Aspect 58: A method according to any one of Aspects 55 to 57, whereinthe first prism includes at least one chamfered edge, and wherein thesecond prism includes at least one chamfered edge.

Aspect 59: A method according to any one of Aspects 55 to 58, whereinthe first prism includes at least one edge with a light-absorbingcoating, wherein the second prism includes at least one edge with thelight-absorbing coating.

Aspect 60: A method according to any one of Aspects 55 to 59, whereinthe first path is a path of the first light before the first lightenters the first prism, wherein the second path is a path of the secondlight before the second light enters the second prism.

Aspect 61: A method according to any one of Aspects 55 to 60, whereinthe first prism includes a first reflective surface configured toreflect the first light, wherein the second prism includes a secondreflective surface configured to reflect the second light.

Aspect 62: A method according to any one of Aspects 55 to 61, whereinthe first path is a path of the first light after the first light entersthe first prism but before the first reflective surface reflects thefirst light, wherein the second path is a path of the second light afterthe second light enters the second prism but before the secondreflective surface reflects the second light.

Aspect 63: A method according to any one of Aspects 50 to 62, whereinthe first image and the second image are captured contemporaneously.

Aspect 64: A method according to any one of Aspects 50 to 63, whereinthe first light redirection element is fixed relative to the first imagesensor, wherein the second light redirection element is fixed relativeto the second image sensor.

Aspect 65: A method according to any one of Aspects 50 to 64, wherein afirst planar surface of the first image sensor faces a first direction,wherein a second planar surface of the second image sensor faces asecond direction that is parallel to the first direction.

Aspect 66: A method according to any one of Aspects 50 to 64, furthercomprising: modifying at least one of the first image and the secondimage using a brightness uniformity correction

Aspect 67: An apparatus for digital imaging, the apparatus comprisingmeans for performing operations according to any of aspects to 31 to 66.

Aspect 68: A computer-readable storage medium storing instructions that,when executed, cause one or more processors to perform operationsaccording to any of aspects to 31 to 66.

Aspect 69: An apparatus for digital imaging, the apparatus comprising: amemory; and one or more processors configured to: receive a first imageof a scene captured by a first image sensor, wherein a first lightredirection element is configured to redirect a first light from a firstpath to a redirected first path toward the first image sensor, whereinthe first image sensor is configured to capture the first image based onreceipt of the first light at the first image sensor; receive a secondimage of the scene captured by a second image sensor, wherein a secondlight redirection element is configured to redirect a second light froma second path to a redirected second path toward the second imagesensor, wherein the second image sensor is configured to capture thesecond image based on receipt of the second light at the second imagesensor, wherein a virtual extension of the first path beyond the firstlight redirection element intersects with a virtual extension of thesecond path intersect beyond the second light redirection element;modify at least one of the first image and the second image using aperspective distortion correction; and generate a combined image fromthe first image and the second image in response to modification of theat least one of the first image and the second image using theperspective distortion correction, wherein the combined image includes acombined image field of view that is larger than at least one of a firstfield of view of the first image and a second field of view of thesecond image.

Aspect 70: An apparatus according to Aspect 69, wherein the one or moreprocessors configured to perform operations according to any of aspectsto 32 to 49 or 51 to 66.

Aspect 71: A method for digital imaging, the method comprising:receiving a first image of a scene captured by a first image sensor,wherein a first light redirection element is configured to redirect afirst light from a first path to a redirected first path toward thefirst image sensor, wherein the first image sensor is configured tocapture the first image based on receipt of the first light at the firstimage sensor; receiving a second image of the scene captured by a secondimage sensor, wherein a second light redirection element is configuredto redirect a second light from a second path to a redirected secondpath toward the second image sensor, wherein the second image sensor isconfigured to capture the second image based on receipt of the secondlight at the second image sensor, wherein a virtual extension of thefirst path beyond the first light redirection element intersects with avirtual extension of the second path intersect beyond the second lightredirection element; modifying at least one of the first image and thesecond image using a perspective distortion correction; and generating acombined image from the first image and the second image in response tomodification of the at least one of the first image and the second imageusing the perspective distortion correction, wherein the combined imageincludes a combined image field of view that is larger than at least oneof a first field of view of the first image and a second field of viewof the second image.

Aspect 72: A method according to Aspects 71, further comprisingoperations according to any of aspects to 32 to 49 or 51 to 66.

What is claimed is:
 1. An apparatus for digital imaging, the apparatuscomprising: a memory; and one or more processors configured to: receivea first image of a scene captured by a first image sensor, wherein afirst light redirection element is configured to redirect a first lightfrom a first path to a redirected first path toward the first imagesensor, wherein the first image sensor is configured to capture thefirst image based on receipt of the first light at the first imagesensor; receive a second image of the scene captured by a second imagesensor, wherein a second light redirection element is configured toredirect a second light from a second path to a redirected second pathtoward the second image sensor, wherein the second image sensor isconfigured to capture the second image based on receipt of the secondlight at the second image sensor, wherein a virtual extension of thefirst path beyond the first light redirection element intersects with avirtual extension of the second path intersect beyond the second lightredirection element; modify at least one of the first image and thesecond image using a perspective distortion correction; and generate acombined image from the first image and the second image in response tomodification of the at least one of the first image and the second imageusing the perspective distortion correction, wherein the combined imageincludes a combined image field of view that is larger than at least oneof a first field of view of the first image and a second field of viewof the second image.
 2. The apparatus of claim 1, wherein, to modify atleast one of the first image and the second image using the perspectivedistortion correction, the one or more processors are configured to:modify the first image from depicting a first perspective to depicting acommon perspective using the perspective distortion correction; andmodify the second image from depicting a second perspective to depictingthe common perspective using the perspective distortion correction,wherein the common perspective is between the first perspective and thesecond perspective.
 3. The apparatus of claim 1, wherein, to modify atleast one of the first image and the second image using the perspectivedistortion correction, the one or more processors are configured to:identify depictions of one or more objects in image data of at least oneof the first image and the second image; and modify the image data atleast in part by projecting the image data based on the depictions ofthe one or more objects.
 4. The apparatus of claim 1, wherein, togenerate the combined image from the first image and the second image,the one or more processors are configured to: align a first portion ofthe first image with a second portion of the second image; and stitchthe first image and the second image together based on the first portionof the first image and the second portion of the second image beingaligned.
 5. The apparatus of claim 1, further comprising: the firstimage sensor; the second image sensor; the first light redirectionelement; and the second light redirection element.
 6. The apparatus ofclaim 1, wherein: the first light redirection element includes a firstreflective surface, wherein, to redirect the first light toward thefirst image sensor, the first light redirection element uses the firstreflective surface to reflect the first light toward the first imagesensor; and the second light redirection element includes a secondreflective surface, wherein, to redirect the second light toward thesecond image sensor, second light redirection element uses the secondreflective surface to reflect the second light toward the second imagesensor.
 7. The apparatus of claim 1, wherein: the first lightredirection element includes a first prism configured to refract thefirst light; and the second light redirection element includes a secondprism configured to refract the second light.
 8. The apparatus of claim7, wherein the first prism and the second prism are contiguous.
 9. Theapparatus of claim 8, wherein a bridge joins a first edge of the firstprism and a second edge of the second prism, wherein the bridge isconfigured to prevent reflection of light from at least one of firstedge of the first prism and the second edge of the second prism.
 10. Theapparatus of claim 7, wherein the first prism includes at least onechamfered edge, and wherein the second prism includes at least onechamfered edge.
 11. The apparatus of claim 7, wherein the first prismincludes at least one edge with a light-absorbing coating, wherein thesecond prism includes at least one edge with the light-absorbingcoating.
 12. The apparatus of claim 7, wherein the first path is a pathof the first light before the first light enters the first prism,wherein the second path is a path of the second light before the secondlight enters the second prism.
 13. The apparatus of claim 7, wherein thefirst prism includes a first reflective surface configured to reflectthe first light, wherein the second prism includes a second reflectivesurface configured to reflect the second light.
 14. The apparatus ofclaim 13, wherein the first path is a path of the first light after thefirst light enters the first prism but before the first reflectivesurface reflects the first light, wherein the second path is a path ofthe second light after the second light enters the second prism butbefore the second reflective surface reflects the second light.
 15. Theapparatus of claim 1, wherein the first image and the second image arecaptured contemporaneously.
 16. The apparatus of claim 1, wherein thefirst light redirection element is fixed relative to the first imagesensor, wherein the second light redirection element is fixed relativeto the second image sensor.
 17. The apparatus of claim 1, wherein afirst planar surface of the first image sensor faces a first direction,wherein a second planar surface of the second image sensor faces asecond direction that is parallel to the first direction.
 18. Theapparatus of claim 1, wherein the one or more processors are configuredto: modify at least one of the first image and the second image using abrightness uniformity correction.
 19. A method for digital imaging, themethod comprising: receiving a first image of a scene captured by afirst image sensor, wherein a first light redirection element redirectsa first light from a first path to a redirected first path toward thefirst image sensor, wherein the first image sensor captures the firstimage based on receipt of the first light at the first image sensor;receiving a second image of the scene captured by a second image sensor,wherein a second light redirection element redirects a second light froma second path to a redirected second path toward the second imagesensor, wherein the second image sensor captures the second image basedon receipt of the second light at the second image sensor, wherein avirtual extension of the first path beyond the first light redirectionelement intersects with a virtual extension of the second path intersectbeyond the second light redirection element; modifying at least one ofthe first image and the second image using a perspective distortioncorrection; and generating a combined image from the first image and thesecond image in response to modification of the at least one of thefirst image and the second image using the perspective distortioncorrection, wherein the combined image includes a combined image fieldof view that is larger than at least one of a first field of view of thefirst image and a second field of view of the second image.
 20. Themethod of claim 19, wherein modifying at least one of the first imageand the second image using the perspective distortion correctionincludes: modifying the first image from depicting a first perspectiveto depicting a common perspective using the perspective distortioncorrection; and modifying the second image from depicting a secondperspective to depicting the common perspective using the perspectivedistortion correction, wherein the common perspective is between thefirst perspective and the second perspective.
 21. The method of claim19, wherein modifying at least one of the first image and the secondimage using the perspective distortion correction includes: identifyingdepictions of one or more objects in image data of at least one of thefirst image and the second image; and modifying the image data byprojecting the image data based on the depictions of the one or moreobjects.
 22. The method of claim 19, wherein: the first lightredirection element includes a first reflective surface, wherein, toredirect the first light toward the first image sensor, the first lightredirection element uses the first reflective surface to reflect thefirst light toward the first image sensor; and the second lightredirection element includes a second reflective surface, wherein, toredirect the second light toward the second image sensor, second lightredirection element uses the second reflective surface to reflect thesecond light toward the second image sensor.
 23. The method of claim 19,wherein: the first light redirection element includes a first prismconfigured to refract the first light; and the second light redirectionelement includes a second prism configured to refract the second light.24. The method of claim 23, wherein the first prism and the second prismare contiguous.
 25. The method of claim 24, wherein a bridge joins afirst edge of the first prism and a second edge of the second prism,wherein the bridge is configured to prevent reflection of light from atleast one of first edge of the first prism and the second edge of thesecond prism.
 26. The method of claim 23, wherein the first path is apath of the first light before the first light enters the first prism,wherein the second path is a path of the second light before the secondlight enters the second prism.
 27. The method of claim 23, wherein thefirst prism includes a first reflective surface configured to reflectthe first light, wherein the second prism includes a second reflectivesurface configured to reflect the second light.
 28. The method of claim27, wherein the first path is a path of the first light after the firstlight enters the first prism but before the first reflective surfacereflects the first light, wherein the second path is a path of thesecond light after the second light enters the second prism but beforethe second reflective surface reflects the second light.
 29. The methodof claim 19, wherein the first image and the second image are capturedcontemporaneously.
 30. The method of claim 19, further comprising:modifying at least one of the first image and the second image using abrightness uniformity correction.